How to Fix the SEO Crawl Budget Problem in WordPress

Crawl budget is the limit Google sets on how many pages it will crawl on your website.

Apart from your posts and pages, WordPress generates several extra URLs that search engines can crawl and index. This means Google may spend your crawl budget on unnecessary pages instead of your important content.

In this article, we’ll show you how to optimize your crawl budget in WordPress, so Google finds your important content first.

Quick Answer: How to Fix the Crawl Budget in WordPress

Blocking unnecessary URLs with your SEO plugin’s crawl cleanup settings or robots.txt ensures your crawl budget isn’t wasted.

Most small websites may never run out of crawl budget. However, WordPress creates many more URLs than you publish. Google may end up wasting crawl budget on these URLs, and your important pages may not get crawled quickly or frequently.

Use the quick links below to jump straight to the fix you need:

What Is Crawl Budget and How to Check It?

Crawl budget is the number of pages a search engine like Google will crawl on your website within a specific timeframe. There are two main factors:

• Crawl Capacity: How many requests Google can make without crashing or slowing down your server. If your server is fast and reliable, your capacity is high.
• Crawl Demand: How much Google wants to crawl your site based on its popularity, overall quality, and how often you publish fresh content.

When low-value pages or slow servers eat up your crawl budget, Google crawls less, and it takes longer to index your new content.

It’s worth taking a moment to examine your Crawl Stats report in Google Search Console. It tells you when and how often Googlebot visits your website. Simply open Google Search Console and go to Settings » Crawl stats » Open Report to pull it up.

Once you’re in the report, Google breaks down crawl requests in four ways. Here is what each one shows:

• By response. How your server answered each request, including ‘OK (200)’, ‘Not found (404)’, redirects, and server errors (5XX).
• By file type. What Googlebot downloaded across HTML pages, images, CSS, and JavaScript.
• By purpose. ‘Discovery’ means Google is crawling a URL for the first time, while ‘Refresh’ means it is re-crawling pages it already knows.
• By Googlebot type. Shows which crawler made the request. A high ‘Smartphone’ share is normal, since Google mostly crawls the mobile version of your site.

Click any row in these reports to view the specific URLs and their individual crawl stats. This data will also show you exactly which fixes your site needs:

• A large share of ‘Not found (404)’ or ‘Server error (5XX)’ responses means dead URLs are consuming requests. You can solve this by cleaning up your redirect chains and fixing broken links.
• A big ‘Syndication’ number means Google is spending requests on feeds instead of your content. You can easily stop this waste by blocking those unnecessary URLs.
• If your ‘Discovery’ crawl stat remains high even when you aren’t publishing new content, Google is likely wasting time on auto-generated URLs, like sorting and filter pages. You can stop this by applying the same robots.txt blocking rules.
• Slow average response times or repeated server errors cap your whole budget. This is a clear sign that it is time to speeding up your server.

Finally, check the ‘Discovered – currently not indexed’ metric in your Page Indexing report. Google uses this when it delays a crawl to avoid crashing your server. A high count here is a direct indicator that your site’s crawling is capped.

Block Crawl-Wasting URLs With robots.txt or Your SEO Plugin

Most wasted crawl budget comes from URLs that WordPress and your plugins generate automatically, not from the articles you write.

These unnecessary URLs include feeds, comment reply links, internal search results, tracking parameters from your campaigns and ads, and WooCommerce filter parameters.

Before you start disallowing URLs, keep two things in mind:

• Never disallow CSS or JS. Google needs to load these files to display your pages properly.
• Don’t block a URL you plan to noindex. A noindex tag asks Google to keep a page out of search results. However, Google has to crawl the WordPress page to see that tag, so blocking the URL in robots.txt means Google never reads it.

Write Your robots.txt Rules by Hand

The robots.txt file is a plain text file that lives at the root of your website. It tells search engine bots which URLs they should not visit.

You can edit your robots.txt file manually through your hosting account. However, using the built-in editor in your SEO plugin is much easier.

For example, in All in One SEO, you’ll find it under All in One SEO » Tools » Robots.txt Editor.

Other popular plugins like Yoast SEO and Rank Math offer similar robots.txt editors.

Start by identifying the URL patterns you don’t want crawled, like /?s= for internal search, ?replytocom= appended to post URLs for comment reply links, and /feed/ for RSS feeds.

Before you block your RSS feeds, make sure you don’t need them for a podcast, an automated email newsletter, or feed readers. Once you know they are not being used, add a Disallow rule for each one.

User-agent: *
Disallow: /*?s=
Disallow: /*?replytocom=
Disallow: /*/feed/

Once you’ve saved your rules, verify them with Google Search Console‘s robots.txt report before moving on.

Confirm your new Disallow rules show up there with no syntax errors flagged.

Set Up Crawl Cleanup in Your SEO Plugin

If you’d rather not write robots.txt rules by hand, then your SEO plugin’s crawl cleanup settings can do the same job with simple toggles.

If you’re using AIOSEO, you’ll find it under All in One SEO » Search Appearance » Advanced. Then turn on ‘Crawl Cleanup’. The Crawl Cleanup tool lets you disable unnecessary RSS feeds (like the ones WordPress auto-generates for comments, authors, and attachments), block unwanted bots, and stop crawling of internal search URLs.

Make sure you leave your main site feed on. That is why AIOSEO marks it as ‘not recommended to disable’.

Yoast SEO has the same functionality under Yoast SEO » Settings » Advanced » Crawl optimization, with its own feed-removal and URL-parameter cleanup options.

Once you’ve turned on these options, check your robots.txt file to confirm the new rules actually active.

How to Handle Tracking Parameters

Email campaigns, ads, and marketing tools often add tracking parameters to your URLs (like utm_source, fbclid, or gclid) to see where visitors come from.

AIOSEO’s Query Arg Monitoring shows you which of these query args (the extra values after a question mark in a URL) Googlebot is actually crawling. You’ll find it at the bottom of the same ‘Advanced’ screen in AIOSEO.

Most of the time, you don’t need to block or redirect tracking links.

While AIOSEO offers an ‘Optimize UTM Parameters’ toggle that rewrites these links via redirects, we highly recommend leaving it off so you don’t accidentally break tracking data.

However, if you spot query arguments in your monitor that serve absolutely no purpose, you can safely add them to the block list in this section. Advanced users can also use Regex (regular expressions) to block multiple similar URLs at once.

Clean Up WooCommerce Filter and Cart URLs

WooCommerce stores add their own crawl-wasting parameters on top of everything above, mainly orderby=, filter_, min_price= and max_price=, and add-to-cart=.

You can block the ones with no search value using the same robots.txt approach, or add them as query args in Query Arg Monitoring from the section above.

Blocking add-to-cart= is safe since it triggers an action rather than serving unique content.

Do the same for the sorting and price-range filters, with one exception – a filter that earns search traffic on its own, like a single popular size or color.

Deindex Thin Content – Tag, Date, and Author Archives

Deindexing means removing a page from Google’s search results. You do it by adding a noindex tag, which asks search engines not to show that page when people search.

WordPress automatically creates archive pages that group your posts by tag, date, and author. These archives mostly repeat content that already exists on your site, which is why SEO experts call them ‘thin’ pages.

Here are four archive types that are usually safe to noindex:

• Author archives, if your site has a single writer
• Tag archives that don’t work as real landing pages
• Date archives, which rarely carry any search value on their own
• Attachment pages, which older WordPress sites create for every uploaded image (WordPress 6.4 stopped making them for new installations in November 2023)

You can set each of the archive types to noindex from your SEO plugin’s archive settings. Most WordPress SEO plugins include this setting in their free versions.

In AIOSEO, you’ll find it under All in One SEO » Search Appearance » Archives.

Before you save, double-check you haven’t accidentally noindexed an important page, or an entire post type.

Keep Your XML Sitemap Clean

Your XML sitemap tells Google which URLs you consider important and fresh. By keeping your sitemap clutter-free, you ensure search engines focus on your most valuable content rather than wasting time on paginated archives.

By default, WordPress creates an XML sitemap for your site. For more control, we recommend generating a custom sitemap with an SEO plugin.

A custom sitemap adds last-modified dates and removes the pages that you set to noindex from it. AIOSEO, Yoast SEO, and Rank Math all enable you to create a sitemap in their free versions.

Fix Crawl Errors and Redirect Chains

Dead and redirected URLs still cost you crawl requests because Google only learns a URL is dead by fetching it. On a typical WordPress site, these wasted crawl requests usually come from three main culprits:

• Redirect chains. This happens when URL A redirects to B, which then redirects to C. To fix this, collapse the chain by routing the original URL directly to the final destination with a single 301 (permanent) redirect.
• Soft 404 errors. This occurs when a page is removed, but your server still returns a normal “200 OK” success code. Google will keep crawling the URL because the server hasn’t explicitly said the content is gone. You can fix this by ensuring deleted pages return a proper 404 (Not Found) or 410 (Gone) status code.
• Outdated internal links. Every time Google follows an internal link that leads to a broken page or a redirect, it burns through your crawl budget. You should regularly audit your internal links and either remove them or update them to point directly to the live, final URL

You can also read our guide on how to create a redirect in WordPress, which walks you through the entire process.

Speed Up Your Server to Raise Your Crawl Rate

A fast server encourages Google to crawl more pages, while a struggling server forces it to back off. To improve your response times, you need two things: robust WordPress hosting and an active caching setup.

However, server errors can hurt your crawl budget even more than your page speed. If Googlebot frequently hits timeouts, it drastically cuts back on crawling and is slow to recover.

You can monitor response times and server errors in Google Search Console’s Crawl Stats report. Errors are usually caused by a maxed-out hosting plan or a faulty plugin.

See our guide to fixing 500 internal server errors for help with error logs and hosting limits

Block AI Crawlers That Waste Your Server Resources

AI crawlers like GPTBot, ClaudeBot, and PerplexityBot don’t use your Google crawl budget. However, they hit the same server, and aggressive scraping can slow your site down enough to drag your Google crawl rate down with it.

The quickest solution to fix this issue is crawl cleanup that comes with most SEO plugins. AIOSEO’s Crawl Cleanup, includes bot-blocking settings that stop unwanted AI bots at the WordPress level.

For stronger protection, Cloudflare’s free AI Crawl Control lets you block individual crawlers from its Crawlers tab. It blocks crawlers before they ever reach your hosting server.

However, use caution when configuring your blocklist. Many of these ‘bots’ may be AI search assistants that drive traffic to your site, so blocking them indiscriminately will hurt your overall visibility.

Frequently Asked Questions About WordPress Crawl Budget

Here are the questions we get most often about crawl budget.

Can you set or increase your crawl budget?

There’s no fixed number that you can set. You can raise your crawl capacity with a faster, more reliable server, and you can raise crawl demand with fresh, non-duplicate content that Google actually wants to revisit.

How do I stop Google from crawling the /feed/ URLs showing up as duplicates in Search Console?

Disable the redundant feeds with AIOSEO’s Crawl Cleanup, or add a Disallow rule for your feed URLs to your robots.txt. That content is already indexed elsewhere on your site, so blocking the feeds costs you nothing in terms of organic traffic.

Does the noindex tag save crawl budget?

Not directly, because Google still has to crawl the page to see the tag. Instead, stop Google from crawling via a robots.txt file.

Will a custom 404 page with helpful links improve my crawl budget?

No, it improves user experience though. To ensure crawl budget is not wasted, ensure your server sends the correct 404 or 410 status code for a page that no longer exists.

We hope this article helped you find and fix what’s wasting your crawl budget. Give the changes a few weeks, then check Crawl Stats again to see the improvement.

You may also want to read our ultimate WordPress SEO guide or how to stop search engines from crawling a WordPress site.

If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.