Key takeaways
- Google describes a rendering phase in which Googlebot runs JavaScript, while seogeoaeo.ai reports that most AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, read raw HTML only.
- Content, prices, structured data, links and metadata added by JavaScript can be missing from what AI crawlers receive.
- Server-side rendering or static generation fixes the gap, and the seogeoaeo.ai AI Crawler View lists what is missing for 8 credits a run.
Do AI crawlers run JavaScript? For the most part, no. The AI Crawler View on seogeoaeo.ai, built to test exactly this, states that most AI crawlers, GPTBot, ClaudeBot and PerplexityBot among them, read the HTML a server returns and do not run scripts, which means a page built in the browser can reach them close to empty.
Two crawlers, two different pages
Google handles this with a separate step. Its JavaScript SEO guide describes three phases for every page: crawling, rendering and indexing. Googlebot queues pages for rendering, runs the scripts, and indexes what comes out. A React or Vue site that fills its content on the client can still rank in Google because of that second step.
AI crawlers work differently. None of the vendor pages for GPTBot, ClaudeBot or PerplexityBot describe a rendering step, and the seogeoaeo.ai tool page treats skipping it as the normal case. The result is a split many site owners have not noticed: the same URL can look complete to Google and blank to the bots that feed ChatGPT, Claude and Perplexity.
What goes missing
The gap is not only body text. The AI Crawler View compares the raw HTML with the page rendered in a headless browser and reports anything that exists only after JavaScript runs:
| Element | Why losing it matters for AI answers |
|---|---|
| Main content and headings | Nothing to quote or summarize |
| Prices | Product questions get answered from someone else's page |
| Structured data | Facts in JSON-LD never reach the bot |
| Links | Deeper pages are not discovered |
| Metadata | Titles and descriptions may be generic shell values |
A pricing table injected by a client-side widget is a typical case. It looks fine to every person who visits, yet a bot reading raw HTML sees a placeholder.
Why this belongs on the AI checklist
Most conversations about AI crawlers stop at robots.txt: allow or block GPTBot, allow or block ClaudeBot. That is half the story. A site can welcome every AI bot in robots.txt and still be invisible to them because the words are drawn in by scripts. The reverse also happens: teams spend time on llms.txt files and AI-friendly copy while the copy itself never ships in the first HTML response.
There is a search side to this too. OpenAI says OAI-SearchBot surfaces sites in ChatGPT search answers, and Perplexity says PerplexityBot surfaces sites in its results. If those bots receive an empty shell, allowing them in robots.txt achieves little.
The fix is old and boring
Server-side rendering or static generation puts the content into the HTML the server sends. Most modern web frameworks offer one or both. For an older single-page app, prerendering the public pages is a smaller step than a rewrite. The seogeoaeo.ai tool page recommends the same fix.
How to test your own pages
Run a key URL through the AI Crawler View for 8 credits and read the list of content that only appears after scripts run. Then check access separately with the Robots.txt & AI Crawler Checker, and use the AI Bot Log Analyzer on a slice of your access logs to see which bots actually come and what they fetch. The rendering tool does not compare screenshots or check robots.txt itself, so the three runs answer different questions.
A quick manual check is free: view the page source in your browser, not the inspector, and search for a sentence from your main content. If it is not there, AI crawlers probably are not seeing it either. The tool earns its keep when you need the full list of what is missing, including prices, structured data and links.
Frequently asked questions
Does Googlebot run JavaScript?
Yes. Google's JavaScript SEO guide says Google processes pages in three phases, crawling, rendering and indexing, and queues pages for rendering.
How can I tell if AI crawlers see my content?
View the page source, not the browser inspector, and search for a sentence from your main content. For a full list of what only appears after scripts run, compare raw and rendered HTML with a tool such as the seogeoaeo.ai AI Crawler View.
Sources
- AI Crawler Viewseogeoaeo.ai
- Understand JavaScript SEO basicsGoogle Search Central
- Overview of OpenAI crawlersOpenAI
- Perplexity crawlersPerplexity