An Analysis of AI User-Agents and Real-Time Web Page Rendering for AIVO

At RivalSee, we are dedicated to pioneering research in the AI SEO and AIVO (AI Visibility Optimization) space. To that end, we conducted a technical analysis to determine precisely when and how current-generation AI chat models are fetching and rendering web pages. Understanding these mechanics is critical for any SEO specialist operating on the cutting edge of the industry.

The Experiment Protocol

We initiated the experiment by publishing a new webpage that had not yet been submitted for indexing by any major search engine. This created a controlled environment to test direct URL fetching capabilities. We then prompted five leading AI models to summarize the content of the page.

Data and Observations

Our log analysis produced the following data on user-agent behavior and rendering patterns.

AI Model Web Rendering Analysis

Key Findings for AIVO

AI engines are not loading Javascript when doing realtime search. AI Engines like ChatGPT are only loading the HTML of pages when using their search tools and likely need the page to load quickly. While Google has promised that content rendered using client-side JS would be included when computing its blue link search results, it is not currently included in AI search results for ChatGPT, Gemini, Perplexity or Google AI. If you are building a website, make sure all of your content is available even if Javascript is disabled and images are not loaded.

OpenAI 4o - does not load any URL given in the prompt. Surprisingly, OpenAI 4o refused to load any URL given in the prompt. Instead, the model appears to query Google’s index for the URL string. As our test page was not indexed, the model concluded it did not exist, reinforcing the dependency of some AIs on traditional search engine indexes.

Mostly No Caching for Now. With the exception of Claude Sonnet, no model appeared to cache the web page when it scraped. Instead, each time it was asked to review the web page, it made a new HTML request. While this will likely change in the future, there is an opportunity right now to track when pages are being loaded (We are going to provide a tool for this shortly at RivalSee).

Conclusion

AI chats have changed how people are accessing information and changing the game. The rules that worked for SEO do not work for AIVO. It’s a whole new field and the best recommendation is to start tracking your AI visibility using tools like RivalSee and experiment. On our end, we will continue to experiment and add what we learn to the RivalSee platform so you have the best chances of boosting your AI Visibility.

What To Do Next

Follow us on X at @rivalsee or on Reddit at /u/rivalsee_com to get alerted to more learnings. And of course, sign up for RivalSee to start tracking how your site is performing.

And reach out to us at those accounts if you have any more questions or have ideas for more experiments.