Robots.txt and Crawl Budget for Local Sites
robots.txt directives and crawl budget for local sites.
robots.txt tells search engine crawlers which URLs they may crawl. For most local sites, crawl budget (the number of pages Google will crawl) is not a concern, but robots.txt still matters for preventing accidental blocks. This guide covers robots.txt and crawl budget for local sites. See Technical SEO Audit for the broader context.
What robots.txt does
robots.txt controls crawling, not indexing. A URL blocked by robots.txt may still be indexed if Google discovers it through links — it just will not be crawled, so Google may show it without a description. Google documents robots.txt in Search Central.
Common directives
| Directive | Effect |
|---|---|
| User-agent: * | Applies to all crawlers |
| Disallow: / | Blocks all crawling |
| Disallow: /admin/ | Blocks the /admin/ path |
| Allow: / | Allows all crawling (default) |
| Sitemap: https://…/sitemap.xml | Points to the sitemap |
Crawl budget
Crawl budget is the number of pages Google will crawl on a site within a given timeframe. For small local sites (under 10,000 pages), crawl budget is rarely a concern. For large portfolios, managing crawl budget means: blocking non-indexable URLs (faceted navigation, parameter variants), keeping the sitemap clean, and avoiding duplicate content.
Primary sources
- robots.txt documentation — Google Search Central (accessed 2025-01-15)