
How fast should an SEO crawler crawl your site? We slowed down our own server finding out
An early SEOFix test crawl at about 8 requests per second strained a 2-core Next.js job site. What we changed, what Google and Bing do about crawl rate, and how to throttle any crawler.
2,191 words · 10 min read

