# Can AI crawlers see your site? Test what GPTBot gets without JavaScript

> GPTBot and ClaudeBot were measured fetching JavaScript without running it. A curl test shows what they get from your pages, and how to fix an empty page.

Source: https://seofix.ai/blog/do-ai-crawlers-render-javascript  
Published: 2026-10-19  
Updated: 2026-10-09  
Author: SEOFix Team  
Category: AI search  
Tags: AI agents, Technical SEO, Guides  
Length: 1,693 words, 8 min read

Most AI crawlers don't render JavaScript, as far as anyone has measured. The one large public study, Vercel's analysis of its own network published in December 2024, found that GPTBot and ClaudeBot download JavaScript files but don't execute them, while Googlebot (which also serves Gemini) and Applebot render pages in a browser ([Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler)). OpenAI, Anthropic and Perplexity don't say either way in their crawler docs. So assume GPTBot, ClaudeBot and PerplexityBot see your server's HTML and nothing else, and test what that HTML contains. The test takes two commands and is below.

## Do AI crawlers render JavaScript? What's documented

We checked each vendor's crawler documentation on 8 October 2026 for any statement about rendering.

| Crawler | Runs JavaScript? | Evidence |
| --- | --- | --- |
| GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI) | Not documented. Vercel measured no execution for OpenAI's crawler. | [OpenAI bot docs](https://developers.openai.com/api/docs/bots) say nothing about rendering. [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) |
| ClaudeBot, Claude-SearchBot, Claude-User (Anthropic) | Not documented. Vercel measured no execution for Claude's crawler. | [Anthropic's crawler page](https://support.claude.com/en/articles/8896518) says nothing about rendering. [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) |
| PerplexityBot, Perplexity-User | Not documented. Vercel lists Perplexity among crawlers that don't render. | [Perplexity bot docs](https://docs.perplexity.ai/guides/bots), [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) |
| Googlebot (and Gemini, which uses Google's crawl) | Yes, in headless Chromium, after a queue | [Google](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) |
| Applebot | Yes, sometimes: "Applebot may render the content of your website within a browser" | [Apple](https://support.apple.com/en-us/119829) |

The numbers behind the "no" column come from one source. Vercel counted 569 million GPTBot requests and 370 million Claude requests in a month across its network. JavaScript files were 11.50% of ChatGPT's fetches and 23.84% of Claude's, and Vercel concluded that ["none of the major AI crawlers currently render JavaScript"](https://vercel.com/blog/the-rise-of-the-ai-crawler), naming OpenAI, Anthropic, Meta, ByteDance and Perplexity. Later write-ups, such as [TripleDart's](https://www.tripledart.com/research/why-your-site-is-invisible-to-chatgpt-and-claude), mostly re-cite the same study.

That study is almost two years old, and any vendor could add rendering without announcing it. We haven't found anything newer that measures it directly. The practical conclusion doesn't depend on who's right, though. Content in the server HTML works for every crawler, rendering or not. Content that only appears after JavaScript runs works only for the crawlers that render.

Google renders too, but not straight away. Pages wait in a render queue that ["may stay on this queue for a few seconds, but it can take longer than that"](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), and Google's own advice is that ["server-side or pre-rendering is still a great idea"](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) because ["not all bots can run JavaScript"](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

## Test it: what GPTBot gets from your page

### The 30-second version

Fetch the page with GPTBot's user agent (from [OpenAI's docs](https://developers.openai.com/api/docs/bots)) and look for the things that matter:

```bash
UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
URL='https://example.com/some-page'

curl -s -A "$UA" "$URL" > raw.html
grep -o -i '<title>[^<]*</title>' raw.html
grep -o -i '<link rel="canonical"[^>]*>' raw.html
grep -o -i '<h1[^>]*>[^<]*' raw.html
grep -c -i '<a ' raw.html
```

If the title is generic, the H1 is missing, or the link count is tiny compared with what you see in the browser, AI crawlers are getting a shell. A default Vite + React app is the extreme case. Its `index.html` body is this and nothing else ([create-vite template](https://unpkg.com/create-vite/template-react-ts/index.html)):

```html
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
```

That's everything a non-rendering crawler gets from every route.

Send GPTBot's user agent rather than curl's default, because some servers, CDNs and frameworks answer bots differently from browsers. If you'd rather test with a normal user agent first, compare both: a difference between the two is a finding in itself.

### Compare raw with rendered

`View source` in the browser shows the raw HTML (what non-rendering crawlers get). DevTools' **Elements** panel shows the DOM after JavaScript ran (what Googlebot ends up indexing). Comparing the two by eye works for one page. For anything more, let headless Chrome dump the rendered DOM and compare the important fields with a small script:

```bash
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --headless --dump-dom "$URL" > rendered.html   # Linux: google-chrome --headless --dump-dom
python3 seo_diff.py raw.html rendered.html
```

```python
# seo_diff.py: compare title, canonical, H1 count, visible words and links
import sys
from html.parser import HTMLParser

class Facts(HTMLParser):
    def __init__(self):
        super().__init__()
        self.title, self.canonical, self.h1, self.links, self.words = "", None, 0, 0, 0
        self._skip, self._in_title = 0, False
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag in ("script", "style", "noscript", "template"): self._skip += 1
        elif tag == "title": self._in_title = True
        elif tag == "h1": self.h1 += 1
        elif tag == "a" and a.get("href"): self.links += 1
        elif tag == "link" and a.get("rel") == "canonical": self.canonical = a.get("href")
    def handle_endtag(self, tag):
        if tag in ("script", "style", "noscript", "template"): self._skip = max(0, self._skip - 1)
        elif tag == "title": self._in_title = False
    def handle_data(self, data):
        if self._in_title: self.title += data.strip()
        elif not self._skip: self.words += len(data.split())

def facts(path):
    f = Facts(); f.feed(open(path, encoding="utf-8", errors="replace").read()); return f

raw, rendered = facts(sys.argv[1]), facts(sys.argv[2])
for field in ("title", "canonical", "h1", "words", "links"):
    r, d = getattr(raw, field), getattr(rendered, field)
    print(f"{field:10} raw={r!r:45.45} rendered={d!r:45.45} {'' if r == d else '<-- differs'}")
```

Here is the output for one of our own help pages on seofix.ai, a Next.js site rendered on the server, run on 8 October 2026:

```
title      raw='JavaScript rendering and AI crawler checks · rendered='JavaScript rendering and AI crawler checks ·
canonical  raw='https://seofix.ai/help/javascript-rendering' rendered='https://seofix.ai/help/javascript-rendering'
h1         raw=1                                             rendered=1
words      raw=1143                                          rendered=1142                                          <-- differs
links      raw=94                                            rendered=94
```

A one-word difference is noise. On a client-rendered page you'd see raw word and link counts close to zero and a rendered page with hundreds of each.

Treat small differences with judgement. A cookie banner or a "last seen" widget changing the word count by 20 is fine. A missing H1, a different canonical, or a raw title like `Vite + React + TS` on every URL is not.

## What to fix first

Not everything has to be in the server HTML. In order of what costs you most when it's missing:

1. **`<title>`****, meta description, canonical and meta robots.** If these are set from client code, non-rendering crawlers see your fallback values on every page, and a canonical that changes after rendering sends Google two different signals.
2. **The main content.** The article text, the product description, the job listing. This is what an AI assistant would quote.
3. **Internal links.** A crawler that doesn't run JavaScript can only follow links that exist as `<a href>` in the raw HTML. Navigation built with `onClick` handlers is invisible to it.
4. **Structured data.** JSON-LD injected by a tag manager is not in the raw HTML.

Interactive parts (filters, carousels, comment widgets, "like" buttons) can stay client-side.

## Framework fixes

### Next.js App Router

In the App Router, ["by default, layouts and pages are Server Components"](https://nextjs.org/docs/app/getting-started/server-and-client-components), and even Client Components are prerendered to HTML on the first load. Most Next.js apps fail the test for one of two reasons.

**Data fetched in** **`useEffect`****.** A page marked `'use client'` that loads its content after mount ships an empty state in the HTML. Fetch in a Server Component and pass the data down:

```typescript
// app/jobs/[slug]/page.tsx (Server Component: no 'use client')
import { getJob } from '@/lib/jobs'
import ApplyButton from './apply-button' // 'use client' lives in that file only

export default async function JobPage({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params
  const job = await getJob(slug)
  return (
    <main>
      <h1>{job.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: job.descriptionHtml }} />
      <ApplyButton jobId={job.id} />
    </main>
  )
}
```

**Metadata set in client code.** Use the `metadata` export or `generateMetadata`. Next.js says both are ["only supported in Server Components"](https://nextjs.org/docs/app/api-reference/functions/generate-metadata), because metadata is resolved on the server so it can go into the initial HTML.

When `generateMetadata` is slow or dynamic, Next.js can **stream** metadata: it sends the page first and appends the tags to `<body>` later. For bots it recognises as "HTML-limited", it waits and puts them in `<head>` instead ([Next.js](https://nextjs.org/docs/app/api-reference/functions/generate-metadata)). The default list is matched on the user agent, and as of 8 October 2026 the list on Next.js's canary branch includes Bingbot, Applebot, Google's special-purpose crawlers and the social preview bots, but not GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot ([source](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts)). Those crawlers still get the tags in the raw HTML, just in `<body>`. Nobody documents whether they read them there. If you'd rather not find out, the `htmlLimitedBots` option overrides the list, and Next.js's documented way to turn streaming off entirely is:

```typescript
// next.config.ts
import type { NextConfig } from 'next'

const config: NextConfig = {
  htmlLimitedBots: /.*/,
}

export default config
```

Next.js warns that this ["could lead to longer response times"](https://nextjs.org/docs/app/api-reference/functions/generate-metadata), so check whether your metadata streams at all before you change it. Static pages resolve metadata at build time and put it in `<head>` anyway. Our own help page above is one of those.

### Next.js Pages Router

Use [`getStaticProps`](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-props) (static generation) or `getServerSideProps` for the content. A page that renders a spinner and calls an API in `useEffect` has the same problem as a client-only SPA.

### React single-page apps (Vite, Create React App)

You have three options, from least to most work:

1. **Prerender the public routes at build time.** React's `prerender` from `react-dom/static` ["renders a React tree to a static HTML string"](https://react.dev/reference/react-dom/static/prerender). A small build script can render each marketing or content route to its own `index.html`, which the client bundle then hydrates.
2. **Add server-side rendering.** Vite has a [server-side rendering guide](https://vite.dev/guide/ssr) for wiring an SSR entry into your server.
3. **Move content routes to a framework that renders on the server.** Often the cleanest option when most of the site is content and only the app part needs to be a SPA.

A prerendering service that serves rendered HTML only to bot user agents also works, but you then have two versions of each page to keep in sync, and the bot list has to include the AI crawlers. If your site was built with an AI app builder such as Lovable, Bolt or v0, [Lovable SEO and other vibe-coded sites](https://seofix.ai/blog/lovable-seo) covers what breaks on those stacks and how to fix it.

## Checking a whole site

The two-command test tells you about one URL. Sites usually fail by template: every product page, every job page, every blog post at once. So test one URL per template, not one URL per site.

SEOFix audits the raw HTML of every page, which is what non-rendering crawlers read, so a client-rendered template shows up as missing titles, H1s or links across that template. On top of that, a full audit can run a render sample, `JS_DEPENDENT_CONTENT`: up to 20 of the shallowest pages are loaded in headless Chromium and compared with the raw HTML on title, canonical, H1 count, word count and link count. Two caveats, both from our [JavaScript rendering help page](https://seofix.ai/help/javascript-rendering):

- The render sample is an optional crawler feature. It doesn't run in previews or rechecks, and when it isn't enabled an audit has no `JS_DEPENDENT_CONTENT` results. **No issue doesn't prove a page renders without JavaScript.**
- It doesn't cover content that needs third-party scripts, a login, user interaction, or more than about 2 seconds after load to appear.

The flag thresholds and the fix for each field are in [AI visibility and rendering checks](https://seofix.ai/help/checks-ai-and-rendering), and [How the crawler works](https://seofix.ai/help/how-the-crawler-works) explains why SEOFixBot doesn't run JavaScript during the normal crawl. If AI crawlers can't reach your pages at all, rendering is the second question. The first is your robots.txt and CDN, covered in [GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot: what to block and allow](https://seofix.ai/blog/block-gptbot-robots-txt).

## FAQ

### Does ChatGPT see content loaded with JavaScript?

Based on the only published measurement, no for OpenAI's crawlers: Vercel saw GPTBot fetch JavaScript files without executing them ([Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler)). OpenAI's docs don't say. Make the content you want cited part of the server HTML.

### Is Googlebot affected the same way?

No. Googlebot renders JavaScript in headless Chromium, though rendering can be delayed by a queue ([Google](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)). A client-rendered site can rank in Google and still be invisible to GPTBot and ClaudeBot.

### Is server-side rendering required for AI search?

Strictly, the content has to be in the HTML response. Server-side rendering, static generation and build-time prerendering all get it there. Which one fits depends on how often the content changes.

---

Want to see which of your templates depend on JavaScript? [Run a free audit](https://seofix.ai/): SEOFix reads the HTML your server returns, the same HTML AI crawlers get.