Blocked pages in your report

When a firewall or bot challenge answers instead of your page, SEOFix records the page as Blocked by firewall, not as broken, and leaves it out of the health score. To audit those pages, allowlist SEOFixBot with your site's crawler secret (see Let SEOFix through your firewall) and run the audit again.

#How SEOFix detects a challenge

A page counts as blocked when it answers 403, 429 or 503 and the response carries a known firewall signature:

Anything else is treated as your server's real answer. A plain 403 from an unrecognised firewall, for example, is reported as an HTTP error on that page.

A blocked page is not retried and not parsed, so SEOFix runs no page checks on it. A challenge answered with 429 or 503 still makes SEOFixBot slow down for that host.

#What the report shows

• Blocked in the numbers strip at the top of the report, with "by a firewall" underneath.
• A Blocked by firewall issue (severity warning) per site section, where a section is the first path segment (/jobs, /blog, or / for the root). Its details give the number of blocked pages, the provider and up to 5 sample URLs.
• A banner A firewall blocked N% of this audit when blocked and skipped pages are more than 20% of the audit. It links to How to allowlist SEOFix.

Blocked pages do not count against your health score: the score is computed over the pages SEOFix could check.

When pages were blocked, the link graph is incomplete, so SEOFix skips the checks that depend on it: orphan pages, pages with only one incoming link, and redirects with no incoming links. Those checks would otherwise report false positives.

In the API, GET /v1/crawls/{id}/report returns:

{
  "totals": {"pages": 1200, "blocked": 340},
  "blocked": {
    "count": 340,
    "skipped": 812,
    "providers": {"cloudflare": 340},
    "by_section": {"/jobs": {"blocked": 50, "skipped": 812}, "/blog": {"blocked": 290, "skipped": 0}},
    "sample_urls": ["https://example.com/blog/a", "…"]
  },
  "coverage_warning": {
    "code": "blocked_by_firewall",
    "blocked_ratio": 0.573,
    "message": "1152 of 2012 pages were blocked by a firewall challenge, so this audit does not cover them. Allowlist SEOFix and re-run."
  }
}

coverage_warning is null when 20% or less was blocked. stats.blocked_pages also holds the count. Each page row (GET /v1/crawls/{id}/pages) has blocked_by, the provider name or null.

#When SEOFix stops crawling a blocked section

SEOFix stops spending requests where every answer is a challenge:

• Per section. Once SEOFixBot has fetched 50 pages in a section and every one of them was blocked, it skips the rest of that section's URLs. They count as skipped (blocked.skipped and blocked.by_section), not as crawled pages.
• Whole audit. Once it has fetched 50 pages in total and every one was blocked, the audit stops.

A section with at least one page that got through keeps being crawled.

#Get full coverage

1. Verify the site. SEOFixBot sends the crawler secret header only for verified sites. See Verify site ownership.
2. Allowlist the crawler secret in your firewall:
   • Cloudflare: Connect Cloudflare creates the rule, or add it by hand.
   • Other firewalls: add a rule that skips challenges for requests with the X-SEOFix-Verify header set to your crawler secret. See Let SEOFix through your firewall.
3. Test the rule with Check my setup (site page → Cloudflare → Do it myself).
4. Run the audit again with Run audit now.

On Cloudflare's Free plan, Bot Fight Mode cannot be skipped by a rule. If pages are still blocked after the rule is in place, turn Bot Fight Mode off while the audit runs.

If you audit a site you don't control, you can't add the rule. Ask the site owner, or accept partial coverage.

#Related

• Let SEOFix through your firewall
• Connect Cloudflare
• SEOFixBot