If you look at your server logs rather than your analytics, you’ll notice something striking. A large share of the requests hitting your WordPress site come from bots, not people. Search engine crawlers, uptime monitors, SEO tools, AI crawlers, content scrapers, vulnerability scanners and login brute forcers all visit constantly.
Analytics tools mostly filter these out, so many site owners have no idea how much bot traffic they receive. But your server feels every request. I’ve worked on sites where bots accounted for most of the server load, slowing the site for real customers and pushing hosting costs up.
Managing bots well isn’t about blocking all of them. Some bots are essential to your business. The goal is to welcome the good ones, limit the wasteful ones and block the hostile ones.
The three kinds of bots
Good bots
These bots provide value to you:
- Search engine crawlers, like Googlebot and Bingbot, which index your site so people can find it.
- Uptime and performance monitors you’ve set up yourself.
- Social media preview bots that fetch your page title and image when someone shares a link.
- Feed readers and legitimate aggregators.
- Payment and integration services that send webhooks or check your site.
Blocking these by mistake can hurt search visibility, break link previews or stop integrations working.
Grey-area bots
These aren’t malicious, but whether they help you depends on your goals:
- SEO tool crawlers that build backlink and keyword databases.
- AI crawlers that collect content for training models or for AI-powered search answers.
- Archive crawlers that preserve copies of web pages.
- Price comparison and research crawlers.
They consume resources, and some may use your content in ways you don’t want. You get to decide whether to allow them.
Bad bots
These bots work against you:
- Brute force bots trying passwords on wp-login.php and xmlrpc.php.
- Vulnerability scanners probing for outdated plugins and exposed files.
- Content scrapers copying your articles to republish elsewhere.
- Spam bots submitting comments and forms.
- Card testing bots abusing store checkouts to test stolen credit card numbers.
- Inventory hoarding bots adding limited products to carts.
- Aggressive crawlers that ignore rules and request pages so fast they slow your site.
Bad bots often disguise themselves by pretending to be browsers or even Googlebot.
Step 1: Understand your bot traffic
Start by looking at what’s actually happening. Your server access logs show every request with its user agent. Count requests by user agent:
| <code>awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30</code> |
|---|
Then look at which IPs make the most requests, which URLs are hit most and how many requests return errors. You’ll quickly see whether a particular crawler is unusually aggressive or whether login attacks dominate your traffic.
If you use a CDN or cloud firewall, its dashboard usually shows bot traffic breakdowns too.
Step 2: Verify bots that claim to be important
A user agent is just text; anyone can claim to be Googlebot. Before treating a request as a trusted crawler, verify it. Major search engines support reverse DNS verification: genuine Googlebot IPs resolve to hostnames under googlebot.com or google.com, and the hostname resolves back to the same IP. Google and Bing also publish their crawler IP ranges.
Many cloud firewalls verify known good bots automatically and label them, which saves a lot of manual work. Fake Googlebots that fail verification can be blocked confidently.
Step 3: Use robots.txt for polite bots
The robots.txt file tells well-behaved crawlers which parts of your site they may visit. It’s useful for:
- Keeping crawlers out of areas with no search value, like internal search results or endless filter combinations.
- Disallowing specific crawlers entirely.
Remember the limits. robots.txt is a request, not a lock. Good bots follow it; bad bots ignore it. Never rely on robots.txt to protect anything sensitive, and remember it’s public, so listing secret paths there actually advertises them.
Step 4: Decide on AI crawlers
AI companies run crawlers to gather content, and policies around them have evolved quickly. Common crawler names include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, Google-Extended as a control for Google’s AI use, and CCBot from Common Crawl.
Some companies separate crawlers used for training from those used to fetch pages for AI search answers. That lets you make different choices:
- If you want visibility in AI-powered search and assistants, allow the search-focused crawlers.
- If you don’t want your content used for model training, disallow the training-focused crawlers.
Crawler names and policies change, so check each company’s current documentation and review your choices periodically. Also make sure your firewall or security plugin isn’t blocking crawlers you intend to allow.
Step 5: Rate limit everyone sensibly
Rate limiting caps how many requests a single client can make in a given time. It protects against aggressive crawlers and attacks without needing to identify every bot.
Apply stricter limits to sensitive endpoints:
- wp-login.php and xmlrpc.php: a few requests per minute is plenty for real people.
- Search: internal search can be expensive; limit how fast one client can search.
- Checkout and cart actions: protects against card testing and inventory hoarding.
- REST API and admin-ajax.php: limit abusive patterns while allowing normal use.
You can rate limit in Nginx, in a cloud firewall or with fail2ban rules that ban IPs after repeated abuse. Cloud firewalls are especially effective because they stop traffic before it reaches your server.
Step 6: Block known bad actors
- Block XML-RPC entirely if you don’t use it.
- Challenge suspicious traffic with a cloud firewall’s bot detection, which can present an invisible check that humans pass easily.
- Block fake search engine bots that fail verification.
- Block requests for paths you don’t have, like common vulnerable scripts, at the firewall level to reduce noise.
- Use fail2ban to automatically ban IPs that repeatedly fail logins or trigger many 404s on suspicious paths.
Be careful with broad blocks, like whole countries or hosting networks. They can catch legitimate users, monitoring services and integrations. Test after adding rules.
Step 7: Protect forms and checkouts
Bots target interactive features:
- Forms: use honeypot fields and invisible challenges to stop spam without frustrating people.
- Comments: use an anti-spam service and close comments on old posts if they attract spam.
- Registration: protect sign-up forms from fake accounts.
- WooCommerce checkout: card testing attacks submit many small payments with stolen cards. Rate limit checkout, enable your payment gateway’s fraud tools, require challenges for suspicious traffic and watch for spikes in failed payments.
Card testing in particular can result in fees, chargebacks and problems with your payment provider, so act quickly if you notice it.
Step 8: Deal with scrapers
Content scrapers copy your articles and republish them, sometimes outranking you on their low-quality sites. You can’t stop scraping completely, but you can make it harder:
- Rate limit aggressive clients.
- Block identified scraper IPs and networks.
- Make sure your pages include canonical tags, so copied pages that keep them point back to you.
- Include internal links in your content; scraped copies often keep them, linking back to your site.
- File takedown requests with hosts or search engines when your content is copied wholesale.
Step 9: Keep caching strong
Bots that request public pages are much less harmful when those pages are served from cache. A good page cache or CDN means bot requests rarely reach PHP or the database. Make sure caching covers anonymous traffic properly, and that bots can’t easily bypass it with random query strings. Some caching setups ignore unknown query parameters for this reason.
Monitoring over time
Bot patterns change constantly. Review bot traffic every month or so:
- Which bots are most active?
- Are any new crawlers consuming significant resources?
- Are login attacks increasing?
- Are good bots getting errors because of your rules?
Check Search Console’s crawl stats to make sure Googlebot isn’t being blocked or slowed by your protections.
Frequently asked questions
Will blocking bots hurt my SEO?
Blocking verified search engine crawlers will. Blocking bad bots, scrapers and fake crawlers won’t. Always verify before blocking anything that claims to be a search engine.
Does robots.txt stop bad bots?
No. Malicious bots ignore robots.txt. Use it to guide polite crawlers, and use firewalls, rate limiting and server rules to stop bad ones.
Should I block AI crawlers?
It depends on whether you value visibility in AI tools more than control over how your content is used. You can often allow AI search crawlers while blocking training crawlers. Review the choice as policies evolve.
How do I know if bots are slowing my site?
Look at server logs and resource usage. If a large share of requests come from a few user agents or IPs, or load spikes don’t match human traffic in analytics, bots are likely involved.
Final thoughts
Bots are a permanent part of running a website. Some bring you customers, some quietly consume your server and some are actively trying to break in. Understand your traffic, verify important crawlers, guide polite bots with robots.txt, make clear decisions about AI crawlers, rate limit sensitive endpoints, block the hostile ones and keep caching strong. Your server will spend its time serving real people, which is exactly what it’s for.
