# JavaScript rendering and AI crawler checks

> What SEOFix checks in the raw HTML, how the JavaScript render sample, the AI-crawler checks and the Core Web Vitals sample work, and what they don't cover.

Source: https://seofix.ai/help/javascript-rendering · Category: Audits & crawling · Updated: 2026-10-08

SEOFix audits the HTML your server returns, without running JavaScript, because that is what search engines read first and what most AI crawlers read at all. On top of that, a full audit can render a sample of up to 20 pages in a headless browser and flag content that only appears after JavaScript runs (`JS_DEPENDENT_CONTENT`). Every audit also checks whether robots.txt blocks AI crawlers and whether `/llms.txt` exists.

## Raw HTML: every page

All on-page checks (titles, meta descriptions, headings, canonicals, links, structured data and the rest) run on the HTML of each `200 text/html` response, as served. Content that your JavaScript adds in the browser is not seen by these checks. If your site renders its content client-side, many pages may show missing titles, H1s or links even though they look fine in a browser. The render sample below tells you whether that is the case.

## JavaScript render sample

**Code:** `JS_DEPENDENT_CONTENT` (warning).

**What it does.** After the crawl, SEOFix loads up to 20 of the shallowest pages that answered `200 text/html` (and were not firewall-blocked) in headless Chromium, then compares the rendered page with the raw HTML:

| Field | Flagged when |
|---|---|
| Title | Raw and rendered titles differ. |
| Canonical | Raw and rendered canonical URLs differ. |
| H1 count | Raw and rendered H1 counts differ. |
| Word count | Rendered text has at least 1.5 times the raw word count plus 50 words. |
| Link count | Rendered page has at least 1.5 times the raw link count plus 5 links. |

Each difference is one issue on that URL, with `field`, `raw` and `rendered` values in its details.

**Fix.** Render that content on the server (SSR or static generation) so it is in the raw HTML.

**How pages are rendered.**

- Each page gets a fresh browser context: no cookies or storage carry over between pages.
- The browser only receives your own site's documents, scripts and XHR/fetch responses (same scheme, host and port, or the `www`/apex twin), fetched through the crawler with its normal pacing and robots.txt rules. Images, media, fonts, stylesheets, third-party requests, WebSockets and non-GET requests are blocked.
- A page is read after `DOMContentLoaded` plus up to 2 seconds of settling, with a hard cap of 25 seconds per page. A page that fails or times out is skipped.

**Not covered.**

- Content that needs third-party scripts, a login, cookies, user interaction, or more than about 2 seconds after load to appear.
- Layout and visual rendering (stylesheets and images are not loaded).
- Pages beyond the 20-page sample. The sample shows whether a page type depends on JavaScript; it is not a full rendered crawl.

The render sample is an optional crawler feature. It does not run in previews or rechecks, and when the crawler does not have it enabled, an audit has no `JS_DEPENDENT_CONTENT` results. The absence of this issue is not proof that a page renders without JavaScript.

## AI crawler checks

These run on every audit and on previews.

| Code | Severity | When |
|---|---|---|
| `AI_CRAWLER_BLOCKED` | warning | robots.txt bars one of GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot from your home page. One issue per bot, with the matching `Disallow` rule. |
| `LLMS_TXT_MISSING` | notice | `/llms.txt` does not answer `200` with a non-HTML text file. An HTML page at that address (for example a single-page app's catch-all) does not count. A network error reports nothing. |

### AI crawler response sample

**Code:** `SLOW_FOR_AI_CRAWLERS` (warning).

Many sites and CDNs treat AI user agents differently: rate limits, bot challenges, uncached responses. This optional sample re-fetches up to 20 indexable `200 text/html` pages with GPTBot's user agent plus an `SEOFix-audit (+https://seofix.ai/bot)` marker, paced like the crawl and within robots.txt. A page is flagged when it answers in more than 1 second and more than twice as slow as in the normal crawl. A failed or non-200 probe is skipped.

Like the render sample, it runs only on full audits and only where the crawler has it enabled.

## Core Web Vitals sample

**Codes:** `CWV_POOR_LCP`, `CWV_POOR_INP`, `CWV_POOR_CLS` (warnings).

When enabled on the crawler, a full audit asks Google PageSpeed Insights for the real-user (field) data of up to 20 pages, mobile, one page at a time, for at most 3 minutes in total. A page is flagged when its 75th percentile is worse than:

| Metric | Threshold |
|---|---|
| Largest Contentful Paint | 4 seconds |
| Interaction to Next Paint | 500 ms |
| Cumulative Layout Shift | 0.25 |

Pages without field data in PageSpeed Insights (usually low-traffic pages) get no result. This is not a lab test of your pages.

## Related

- /help/how-the-crawler-works
- /help/reading-your-report
- /help/seofixbot