# AI visibility and rendering checks

> Triggers and fixes for AI crawlers blocked in robots.txt, missing llms.txt, slow responses to AI crawlers, and content that needs JavaScript.

Source: https://seofix.ai/help/checks-ai-and-rendering · Category: Issue reference · Updated: 2026-10-08

These checks cover whether AI assistants (ChatGPT, Claude, Perplexity and others) can read and cite your site. None of them is recheckable; run a full audit after the fix. `AI_CRAWLER_BLOCKED` and `LLMS_TXT_MISSING` run on every audit, previews included. `SLOW_FOR_AI_CRAWLERS` and `JS_DEPENDENT_CONTENT` come from optional samples that run only on full audits where they are enabled, so a missing issue there does not prove the page is fine.

### AI_CRAWLER_BLOCKED — AI crawlers blocked

- **Severity**: warning. **Recheckable**: no (needs a full audit).
- **Trigger**: your `robots.txt` disallows the site's home page (`/`) for one of these user agents: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot`. Each bot is matched against its own group in `robots.txt`, or the `*` group when it has none, so `User-agent: *` with `Disallow: /` flags all five. One issue per blocked bot, reported on your `/robots.txt` URL.
- **Details**: `bot`, `rule` (the matching `Disallow` line, best effort).
- **Why it matters**: AI assistants can't read or cite pages their crawlers are not allowed to fetch.
- **How to fix**: if you want AI visibility, remove the `Disallow: /` for these user agents, or give them their own group that allows the site. If blocking them is a deliberate choice, ignore the issue.

```text
# before
User-agent: GPTBot
Disallow: /

# after: allow GPTBot, keep private paths closed
User-agent: GPTBot
Allow: /
Disallow: /account/
```

### LLMS_TXT_MISSING — llms.txt missing

- **Severity**: notice. **Recheckable**: no (needs a full audit).
- **Trigger**: `/llms.txt` at the site root does not answer `200` with a non-empty, non-HTML body. A `200` with an HTML content type, or a body starting with `<!doctype` or `<html` (a single-page app's catch-all route), counts as missing. If the request fails at the network level, nothing is reported. Reported on the `/llms.txt` URL.
- **Why it matters**: `/llms.txt` gives AI assistants a curated Markdown map of your site's key content.
- **How to fix**: publish a plain-text Markdown file at `/llms.txt`, served as `text/plain` or `text/markdown`, with the site name, a one-line summary and links to key pages. See https://llmstxt.org.

```markdown
# Example Jobs

> Job listings in the UAE, updated daily.

## Key pages

- [All jobs](https://example.com/jobs/): every open role, filterable by city and salary
- [Companies](https://example.com/companies/): employer profiles
- [Salary guide](https://example.com/salaries/): typical salaries by role
```

### SLOW_FOR_AI_CRAWLERS — Slow for AI crawlers

- **Severity**: warning. **Recheckable**: no (needs a full audit).
- **Trigger**: optional sample. SEOFix re-fetches up to 20 indexable `200` HTML pages (shallowest first, only those `robots.txt` allows) with GPTBot's user agent plus an `SEOFix-audit` marker. A page is flagged when that fetch takes more than 1,000 ms and more than twice as long as the same page took in the normal crawl. Failed or non-200 probes are skipped.
- **Details**: `ai_response_ms`, `normal_response_ms`, `bot` (`GPTBot`).
- **Why it matters**: slow or throttled AI fetches mean fewer pages read and cited by AI assistants. CDNs, firewalls and servers often treat AI user agents differently: rate limits, bot challenges or uncached rendering paths.
- **How to fix**: check the rules for AI user agents in your CDN, firewall and server (rate limits, challenges, bot management, cache bypass by user agent) and serve them the same cached response as other visitors.

### JS_DEPENDENT_CONTENT — Content needs JavaScript

- **Severity**: warning. **Recheckable**: no (needs a full audit).
- **Trigger**: optional sample. SEOFix renders up to 20 `200` HTML pages (shallowest first) in a headless browser and compares the result with the raw HTML. One issue per field that only fully exists after JavaScript runs:

| `field` | Flagged when |
| --- | --- |
| `title` | The rendered title differs from the raw HTML title. |
| `canonical` | The rendered canonical differs from the raw HTML canonical. |
| `h1_count` | The number of `<h1>` elements differs. |
| `word_count` | Rendered words ≥ 1.5 × raw words + 50. |
| `link_count` | Rendered links ≥ 1.5 × raw links + 5. |

- **Details**: `field`, `raw`, `rendered`.
- **Why it matters**: many crawlers and most AI bots do not run JavaScript. They see only the raw HTML, so content, links or tags added by scripts are invisible to them.
- **How to fix**: render this content on the server (server-side rendering or static generation) so it is in the raw HTML. Put the `<title>`, canonical and meta tags in the server response rather than setting them from client-side code. You can check what bots get with `curl -s https://example.com/page | grep -i '<h1'`.

## Related

- [Issue reference](https://seofix.ai/help/issue-reference.md)
- [Performance checks](https://seofix.ai/help/checks-performance.md)
- [On-page checks](https://seofix.ai/help/checks-on-page.md)
- [JavaScript rendering](https://seofix.ai/help/javascript-rendering.md)