Crawl budget is one of those SEO terms that gets more attention than it deserves on most sites and less than it deserves on some. For a typical small business site with a few hundred pages, Google can crawl everything comfortably, and crawl budget is irrelevant. For a WooCommerce store with thousands of products and layered navigation filters, or a large directory, it can be the reason new pages take weeks to appear in search results.
This article explains what crawl budget actually is, how to tell whether your site has a problem and the WordPress-specific causes of crawl waste.
What crawl budget means
Google describes crawl budget as the combination of two things.
Crawl capacity limit. How much crawling your server can handle without slowing down. If your server responds quickly and reliably, Google crawls more. If responses are slow or return errors, Google backs off.
Crawl demand. How much Google wants to crawl your site, based on how popular and how frequently updated your pages are, and how many URLs it knows about.
Crawl budget becomes a problem when Google spends its limited crawling on URLs that don’t matter, so the URLs that do matter are crawled less often or discovered late.
Does your site need to care?
Google has said that crawl budget mostly matters for:
- Large sites with more than around a million unique pages that change moderately often
- Medium sites with tens of thousands of pages that change very frequently
- Sites with a large share of pages showing as “Discovered, currently not indexed” in Search Console
For WordPress, I’d add one more group: sites of any size that generate huge numbers of low-value URLs through filters, parameters or plugins. A store with 500 products can still produce hundreds of thousands of crawlable filter combinations.
If your site is small and new pages get indexed within a few days, you probably don’t have a crawl budget problem. Spend your time on content and links instead.
Check Search Console’s Crawl Stats
In Search Console, go to Settings, then Crawl stats. The report shows:
- Total crawl requests per day
- Total download size
- Average response time
- Breakdowns by response code, file type, purpose and Googlebot type
Look at a few things:
- Response codes. A high share of 404s, 301s or 5xx errors means Google is wasting requests.
- Average response time. Rising response times often lead to reduced crawling.
- Purpose. “Discovery” versus “Refresh” shows how much crawling goes to new URLs versus re-checking known ones.
Then look at the Pages report under Indexing. Large numbers of “Discovered, currently not indexed” or “Crawled, currently not indexed” pages, especially if they include important pages, suggest crawling or quality issues.
Look at your server logs
Server logs show exactly which URLs Googlebot requests. They’re the most reliable source for crawl analysis.
Filter access logs for Googlebot’s user agent, verifying with reverse DNS if you need to rule out fake bots, and group requests by URL pattern. A simple command gives a quick overview:
| <code>grep -i "googlebot" access.log | awk '{print $7}' | sed 's/?.*//' | sort | uniq -c | sort -rn | head -30</code> |
|---|
That strips query strings and shows the most-crawled paths. Then run it again without stripping query strings to see which parameters are being crawled.
Log analysis tools like Screaming Frog Log File Analyser make this easier for large sites.
What you’re looking for is mismatch: lots of crawling on URLs you don’t care about, little crawling on URLs you do.
Common WordPress causes of crawl waste
Faceted navigation and filters
WooCommerce layered navigation and filter plugins create URLs like:
- /shop/?filter_color=blue
- /shop/?filter_color=blue&filter_size=m
- /shop/?filter_color=blue&filter_size=m&orderby=price
Every combination of filters, sort orders and pagination becomes a separate URL. Search engines can find millions of them.
Fixes:
- Decide which filter combinations are worth indexing, often none, or a few high-demand ones like a single colour within a category.
- Use canonical tags pointing filtered URLs to the unfiltered category.
- Block crawling of filter and sort parameters in robots.txt when you’re sure they don’t need to be crawled.
- Use filter plugins that load results with JavaScript without creating crawlable links for every combination, or that add nofollow to filter links.
Internal search results
WordPress search URLs look like /?s=keyword. If search results are linked anywhere, or spam bots generate search URLs that get linked from other sites, Google may crawl thousands of them. Search result pages should be noindexed, which most SEO plugins do by default, and you can disallow them in robots.txt:
Tracking and session parameters
Links with utm_source, fbclid, gclid and similar parameters create duplicate URLs. Your canonical tags should point to the clean URL. Avoid using tracking parameters on internal links, which is a surprisingly common mistake in menus and banners.
Add-to-cart and action URLs
WooCommerce add-to-cart links like ?add-to-cart=123 are sometimes crawled. They shouldn’t be. WooCommerce’s default robots.txt handling and nofollow on these links usually prevent it, but check your logs.
Thin archive pages
Date archives, tag archives with one or two posts, author archives on single-author sites and attachment pages all add URLs with little value. Disable those you don’t need. Most SEO plugins can redirect attachment pages to the file or parent post, and WordPress 6.4 and later disables attachment pages by default on new sites.
Feeds
WordPress creates RSS feeds for posts, categories, tags, comments and individual posts. Googlebot crawls some of them. Feeds are generally fine, but on very large sites, removing unnecessary feeds, such as per-post comment feeds, reduces noise.
Redirect chains and broken links
Every redirect is an extra request. Chains of redirects, such as from HTTP to HTTPS to www to a trailing slash, waste crawling. Broken internal links lead to 404s. Fix internal links to point directly to final URLs.
Improve server response
Since crawl capacity depends on how quickly your server responds, performance improvements also increase crawling. Page caching, a CDN and fast hosting all help. If Crawl Stats shows rising response times or server errors, fix those first, because Google reduces crawling when it detects a struggling server.
Help Google find what matters
Reducing waste is half the job. The other half is making important URLs easy to find:
- Keep XML sitemaps accurate, listing only canonical, indexable URLs that return 200 responses.
- Use lastmod dates in sitemaps honestly, so Google can prioritise changed pages.
- Link to important pages from prominent places in your site structure.
- Keep important pages within a few clicks of the homepage.
Robots.txt versus noindex
People often confuse these. Robots.txt controls crawling: disallowed URLs aren’t fetched. Noindex controls indexing: pages are fetched but not added to the index.
For crawl budget, robots.txt is the tool, because it stops requests. But a page disallowed in robots.txt can’t be seen, so Google can’t see a noindex tag on it. If a URL is already indexed and you want it removed, use noindex first, wait until it’s dropped, then disallow it.
A quick audit routine
For sites where crawl efficiency matters, a quarterly routine keeps things under control. Check Crawl Stats for changes in response codes and response time. Pull a sample of Googlebot requests from your logs and group them by URL pattern. Compare the most crawled patterns with the pages you most want indexed. Review the Pages report for growth in “Discovered, currently not indexed”. Then fix the biggest source of waste you find, whether it’s a filter parameter, a plugin generating junk URLs or a set of redirects that should be internal link updates.
Most of the time, one or two specific causes account for the majority of wasted crawling. Fixing those has far more effect than any general tweak.
Plugins that create URLs
Some plugins quietly add crawlable URLs: calendar plugins with endless next-month links, comparison plugins with parameter-based pages, print-friendly versions of every post and translation plugins with incomplete language versions. When you install a plugin that outputs front-end content, crawl the site afterwards and check for new URL patterns.
Don’t over-block
Be careful not to block CSS, JavaScript or image files in robots.txt. Google needs them to render pages properly, and blocking them can affect how your content is understood.
Frequently asked questions
How do I know if Google is crawling my new pages quickly enough?
Use URL Inspection in Search Console on a few new pages. If they’re discovered and indexed within days, crawling is fine. If they sit as “Discovered, currently not indexed” for weeks, investigate.
Will blocking filter URLs hurt my store’s SEO?
Not if the filtered pages weren’t valuable search landing pages. If certain filter combinations get real searches, create proper landing pages for them instead.
Does crawl budget affect rankings?
Not directly. It affects how quickly pages are discovered and updated in the index. Pages that aren’t crawled can’t rank.
Should I use the Search Console crawl rate setting?
Google removed the legacy crawl rate limiter in early 2024. If Googlebot overloads your server, return 503 or 429 responses temporarily, and improve server capacity.
