Status and access checks

These checks are about whether a URL could be fetched at all and what it answered. They are reported on the URL itself. Problems with the links pointing to these URLs are in Links and redirects.

SEOFix fetches with the user agent SEOFixBot/1.0 (+https://seofix.ai/bot), a 15-second timeout, and does not follow redirects automatically (each hop is its own URL).

#BROKEN_PAGE — Broken page

• Severity: error. Recheckable: yes.
• Trigger: a crawled URL answered with a 4xx status (for example 404 or 410), and the answer was not a firewall challenge.
• Details: status_code.
• Why it matters: linked pages that return 4xx waste crawl budget, frustrate visitors and lose any rankings and links they had.
• How to fix: restore the page, 301-redirect it to the closest replacement, or remove the links to it (see BROKEN_INTERNAL_LINK on the linking pages). Use 410 only for content that is gone for good and has no replacement.

# nginx: send a removed page to its replacement
location = /jobs/old-category { return 301 /jobs/new-category; }

#SERVER_ERROR — Server error

• Severity: error. Recheckable: yes.
• Trigger: a crawled URL answered with a 5xx status, and the answer was not a firewall challenge.
• Details: status_code.
• Why it matters: search engines drop pages that keep returning 5xx, and slow down crawling of the whole site.
• How to fix: find the server-side error in your application logs for that URL and fix it. If it only happens under load, the crawl may have hit a capacity limit; SEOFix crawls at a polite rate, so search engines can hit it too. A recheck confirms the page answers again.

#FETCH_FAILED — Page could not be fetched

• Severity: error. Recheckable: yes.
• Trigger: the request failed before any HTTP response: DNS failure, connection refused or reset, TLS error, or a timeout (the crawler's timeout is 15 seconds).
• Details: error, the error name, for example ConnectError, ConnectTimeout or ReadTimeout.
• Why it matters: a page that can't be fetched can't be crawled or indexed.
• How to fix: check DNS records, the TLS certificate (valid, complete chain, matching host) and that the server answers reliably within a few seconds. Timeouts on a few heavy pages usually mean the page is too slow to generate; see SLOW_PAGE.

#BLOCKED_BY_FIREWALL — Blocked by firewall

• Severity: warning. Recheckable: yes.
• Trigger: pages answered 403, 429 or 503 with the signature of a firewall or bot challenge instead of your page: Cloudflare (a cf-mitigated header, or a Cloudflare challenge or block page), Sucuri, DataDome or Akamai. Reported once per top-level section (for example /jobs), on the section's URL. When the first 50 or more fetches in a section are all blocked, SEOFix skips the rest of that section.
• Details: blocked_pages, provider (cloudflare, sucuri, datadome or akamai), sample_urls (up to 5), fix.
• Why it matters: these pages were not checked at all, so the audit is incomplete. Search engine crawlers may be challenged by the same rule.
• How to fix: allowlist SEOFix in your firewall or bot protection. See Let SEOFix through your firewall; the Cloudflare steps are also at https://seofix.ai/docs/cloudflare. Then run a new audit or a recheck.

#PRIVATE_ADDRESS_SKIPPED — Private address skipped

• Severity: notice. Recheckable: yes.
• Trigger: the URL's host resolves to a private or internal IP address. SEOFix never fetches those.
• Why it matters: if a public URL resolves to a private address, visitors and search engines outside your network can't reach it either.
• How to fix: make public hostnames resolve to public IP addresses, and remove links to internal hosts (staging, intranet, localhost) from public pages.

#ROBOTS_BLOCKED — Blocked by robots.txt

• Severity: notice. Recheckable: no (needs a full audit).
• Trigger: SEOFix found the URL, but your robots.txt disallows it for SEOFixBot (the SEOFixBot group, or * when there is none). The URL was not fetched.
• Why it matters: search engines that follow the same rules can't crawl these URLs either. Links from disallowed pages are not counted in the link checks.
• How to fix: if the page should be crawled, remove or narrow the matching Disallow rule. Disallowing /admin, /cart, internal search or /api is normal; ignore the notice for those.

# robots.txt: before, blocks every job page
User-agent: *
Disallow: /jobs

# after: only blocks the internal search under /jobs
User-agent: *
Disallow: /jobs/search

#Related

• Issue reference
• Links and redirects checks
• Verify a fix
• Let SEOFix through your firewall
• SEOFixBot