SEOFixBot, the SEOFix crawler

SEOFixBot is the crawler of SEOFix, an SEO audit service. It visits a site only when a SEOFix user starts an audit of that site, crawls at 2 requests per second by default, and follows robots.txt. To block it, add User-agent: SEOFixBot with Disallow: / to your robots.txt. To report a problem, email [email protected].

#How to recognise it

SEOFixBot sends this user agent:

SEOFixBot/1.0 (+https://seofix.ai/bot)

Each request also carries a Referer header with the page where the link was found, so you can trace the crawl in your logs.

SEOFix does not publish a fixed list of IP addresses for the crawler. Identify it by the user agent.

#Why it visited your site

SEOFixBot runs only when someone asks SEOFix to audit a site:

• a SEOFix user starts an audit in the web app, through the API or through an AI agent connected to SEOFix;
• a visitor runs a free preview from the SEOFix home page (at most 50 pages, about one minute of crawling, and at most one preview per domain per hour);
• a site owner who has proved ownership to SEOFix schedules weekly or daily audits of their own site;
• a page of an audited site links to your site, and SEOFixBot checks that the link works (see Link checks).

If you did not sign up for SEOFix, someone else audited your site, or a site that links to you.

Without proof of ownership, an audit is limited to 500 pages and 3 requests per second. Larger and faster audits need the site owner to verify the site (through Search Console, DNS, a homepage tag or a file on the site).

#How fast it crawls

SEOFixBot slows down on its own:

• It learns your site's normal response time from the first 20 successful pages. If responses get more than twice as slow and at least 300 ms slower, it halves its rate, down to one request every 10 seconds, and speeds up again once your site recovers.
• Network errors, 429 Too Many Requests and 5xx responses also make it back off.

At the default rate, a 10,000-page audit takes about 85 minutes.

#robots.txt

SEOFixBot reads /robots.txt on each host before crawling it and does not fetch pages your rules disallow. Those pages appear in the auditor's report as "Blocked by robots.txt".

• User-agent token: SEOFixBot. Matching is case-insensitive, so seofixbot works too.
• If a group names SEOFixBot, that group applies. Otherwise the User-agent: * group applies.
• Allow, Disallow and the * and $ wildcards are supported.
• robots.txt is read once at the start of each audit. A change applies from the next audit.
• If /robots.txt is missing, answers anything other than 200, or cannot be fetched, SEOFixBot treats every page as allowed.
• Crawl-delay is not supported. SEOFixBot ignores it and paces itself as described above. To slow it down, answer with 429 or 503: it backs off. To stop it, use Disallow.

Block SEOFixBot from the whole site:

User-agent: SEOFixBot
Disallow: /

Block only some paths:

User-agent: SEOFixBot
Disallow: /search
Disallow: /cart/
Disallow: /*?sort=

A group for SEOFixBot replaces your * group for it, so repeat any * rules you also want SEOFixBot to follow.

The audit also reads /robots.txt, /sitemap.xml (and the sitemaps it lists) and /llms.txt, because checking them is part of an SEO audit.

#What it does not do

• It only sends GET requests. It does not submit forms, log in, post comments, add items to carts or create accounts.
• It does not run JavaScript during a normal crawl. It reads the HTML your server returns.
• It does not try to solve CAPTCHAs or bot challenges. A challenged page is reported to the auditor as blocked by a firewall and left alone. If every page in a section is challenged, it stops crawling that section after 50 pages, and it stops the whole audit when the first 50 pages are all challenged.
• It never requests private or internal network addresses.

#Optional samples

Two extra checks exist that are off unless SEOFix enables them. When they run, they use the same pacing and robots.txt rules:

• JavaScript rendering sample: up to 20 pages are loaded in a headless browser to compare rendered and raw content. The browser only loads scripts and data from the same site, with GET requests allowed by robots.txt.
• AI-crawler response sample: up to 20 pages are fetched again with a GPTBot-style user agent that ends in SEOFix-audit (+https://seofix.ai/bot), to see whether AI crawlers get slower answers. You can tell these requests apart by that marker.

#Link checks

When a site being audited links to your site, SEOFixBot requests each linked URL once to check that it works. These checks:

• send one request at a time per host, at least 0.5 seconds apart;
• cover at most 500 distinct external URLs per audit, across all linked sites;
• are single requests to the linked URLs, not a crawl of your site, and do not read your robots.txt.

#Other requests from SEOFix

• When a SEOFix user tries to verify ownership of a site, SEOFix's servers fetch the homepage, /.well-known/seofix-verify.txt or an IndexNow key file at /<key>.txt. These are single requests and may not carry the SEOFixBot user agent.
• A site owner's firewall check loads the start page twice as SEOFixBot.
• When Core Web Vitals sampling is enabled, Google PageSpeed Insights loads up to 20 pages, one at a time. Those requests come from Google, not from SEOFixBot.

#Allowing SEOFixBot on your own site

If you use SEOFix and your firewall blocks the crawler, don't allowlist the user agent: anyone can copy it. Verify your site and allowlist the private X-SEOFix-Verify header instead. See Let SEOFix through your firewall.

#Report a problem

If SEOFixBot causes load on your site or ignores your robots.txt, email [email protected]. Include:

• your domain;
• the time range (with time zone) and a few log lines, including the user agent and the requesting IP addresses;
• what you saw (request rate, paths, errors).

Blocking it in robots.txt takes effect from the next audit.

#Related

• Let SEOFix through your firewall
• Blocked pages in your report