What decides it
Google describes two parts. Crawl capacity is how fast it can fetch without hurting your server, which depends on response times and errors. Crawl demand is how much it wants to fetch, which depends on how popular and how fresh your pages are.
Slow pages, server errors and timeouts reduce capacity. Thin, duplicate and endless parameter URLs reduce the share of the budget spent on pages that matter.
Who should worry
A site with a few thousand pages rarely runs out of crawl budget. It becomes a real constraint for sites with tens of thousands of URLs, faceted navigation that generates endless combinations, or a lot of auto generated pages. Directories and marketplaces fall into this group quickly.
How to spend it well
Serve pages fast, ideally from a cache. Keep the sitemap to indexable 200 URLs. Noindex or remove thin pages instead of linking to them sitewide. Avoid redirect chains. Give each piece of content one URL. Return a real 404 for pages that do not exist, not a soft 404 with a 200 status.
Signs you have a crawl problem
New pages take weeks to appear in Search Console. The Pages report shows many URLs as Discovered, currently not indexed. Crawl stats show Googlebot spending most of its requests on parameter URLs, redirects or 404s. Server response times are high. Any of these suggests the crawler is spending its visits on the wrong pages, or cannot fetch fast enough.
Example
A directory with 40 categories, 35 of them empty, links all 40 from every page. Crawlers spend their visits on empty pages. Linking only the 5 categories with listings sends that attention to pages that can rank.
Questions people ask
- How do I see my crawl activity?
- Google Search Console has a Crawl stats report under Settings. It shows requests per day, response times and status codes.
Related terms
- NoindexNoindex is a directive that asks search engines to keep a page out of their results. It is set with <meta name="robots" content="noindex"> or an X-Robots-Tag HTTP header.
- XML sitemapAn XML sitemap is a file that lists the URLs you want search engines to crawl and index, optionally with the date each page last changed.
- robots.txtrobots.txt is a plain text file at the root of a site that tells crawlers which paths they may and may not fetch. It controls crawling, not indexing.
- 301 redirectA 301 redirect is an HTTP response that tells browsers and search engines a page has moved permanently to a new URL. Search engines move the old URL's signals to the new one.