SEOFix

AI visibility and rendering checks

Triggers and fixes for AI crawlers blocked in robots.txt, missing llms.txt, slow responses to AI crawlers, and content that needs JavaScript.

Updated 8 October 2026View as Markdown

These checks cover whether AI assistants (ChatGPT, Claude, Perplexity and others) can read and cite your site. None of them is recheckable; run a full audit after the fix. AI_CRAWLER_BLOCKED and LLMS_TXT_MISSING run on every audit, previews included. SLOW_FOR_AI_CRAWLERS and JS_DEPENDENT_CONTENT come from optional samples that run only on full audits where they are enabled, so a missing issue there does not prove the page is fine.

AI_CRAWLER_BLOCKED — AI crawlers blocked

  • Severity: warning. Recheckable: no (needs a full audit).
  • Trigger: your robots.txt disallows the site's home page (/) for one of these user agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. Each bot is matched against its own group in robots.txt, or the * group when it has none, so User-agent: * with Disallow: / flags all five. One issue per blocked bot, reported on your /robots.txt URL.
  • Details: bot, rule (the matching Disallow line, best effort).
  • Why it matters: AI assistants can't read or cite pages their crawlers are not allowed to fetch.
  • How to fix: if you want AI visibility, remove the Disallow: / for these user agents, or give them their own group that allows the site. If blocking them is a deliberate choice, ignore the issue.
# before
User-agent: GPTBot
Disallow: /

# after: allow GPTBot, keep private paths closed
User-agent: GPTBot
Allow: /
Disallow: /account/

LLMS_TXT_MISSING — llms.txt missing

  • Severity: notice. Recheckable: no (needs a full audit).
  • Trigger: /llms.txt at the site root does not answer 200 with a non-empty, non-HTML body. A 200 with an HTML content type, or a body starting with <!doctype or <html (a single-page app's catch-all route), counts as missing. If the request fails at the network level, nothing is reported. Reported on the /llms.txt URL.
  • Why it matters: /llms.txt gives AI assistants a curated Markdown map of your site's key content.
  • How to fix: publish a plain-text Markdown file at /llms.txt, served as text/plain or text/markdown, with the site name, a one-line summary and links to key pages. See https://llmstxt.org.
# Example Jobs

> Job listings in the UAE, updated daily.

## Key pages

- [All jobs](https://example.com/jobs/): every open role, filterable by city and salary
- [Companies](https://example.com/companies/): employer profiles
- [Salary guide](https://example.com/salaries/): typical salaries by role

SLOW_FOR_AI_CRAWLERS — Slow for AI crawlers

  • Severity: warning. Recheckable: no (needs a full audit).
  • Trigger: optional sample. SEOFix re-fetches up to 20 indexable 200 HTML pages (shallowest first, only those robots.txt allows) with GPTBot's user agent plus an SEOFix-audit marker. A page is flagged when that fetch takes more than 1,000 ms and more than twice as long as the same page took in the normal crawl. Failed or non-200 probes are skipped.
  • Details: ai_response_ms, normal_response_ms, bot (GPTBot).
  • Why it matters: slow or throttled AI fetches mean fewer pages read and cited by AI assistants. CDNs, firewalls and servers often treat AI user agents differently: rate limits, bot challenges or uncached rendering paths.
  • How to fix: check the rules for AI user agents in your CDN, firewall and server (rate limits, challenges, bot management, cache bypass by user agent) and serve them the same cached response as other visitors.

JS_DEPENDENT_CONTENT — Content needs JavaScript

  • Severity: warning. Recheckable: no (needs a full audit).
  • Trigger: optional sample. SEOFix renders up to 20 200 HTML pages (shallowest first) in a headless browser and compares the result with the raw HTML. One issue per field that only fully exists after JavaScript runs:
field Flagged when
title The rendered title differs from the raw HTML title.
canonical The rendered canonical differs from the raw HTML canonical.
h1_count The number of <h1> elements differs.
word_count Rendered words ≥ 1.5 × raw words + 50.
link_count Rendered links ≥ 1.5 × raw links + 5.
  • Details: field, raw, rendered.
  • Why it matters: many crawlers and most AI bots do not run JavaScript. They see only the raw HTML, so content, links or tags added by scripts are invisible to them.
  • How to fix: render this content on the server (server-side rendering or static generation) so it is in the raw HTML. Put the <title>, canonical and meta tags in the server response rather than setting them from client-side code. You can check what bots get with curl -s https://example.com/page | grep -i '<h1'.

More in Issue reference

Still stuck? Email [email protected] with your site and what you expected to see.