Two small files have an outsized influence on how search engines see your WordPress site. robots.txt tells crawlers where they may and may not go. XML sitemaps tell them which pages exist and which you consider important. Together, they help search engines spend their time on your best content.
When they’re misconfigured, the consequences can be serious. I’ve seen robots.txt files that blocked an entire site from Google after a launch, sitemaps full of redirected and noindexed URLs and robots.txt rules that blocked the CSS and JavaScript Google needs to render pages. All of these were simple mistakes with big effects.
Here’s how both files work in WordPress and how to configure them properly.
Part 1: robots.txt
What robots.txt does
robots.txt is a plain text file at the root of your domain, at example.com/robots.txt. It contains rules for web crawlers, telling them which paths they’re allowed to request.
A basic file looks like this:
- User-agent specifies which crawler the rules apply to. An asterisk means all crawlers.
- Disallow lists paths crawlers shouldn’t request.
- Allow creates exceptions within disallowed paths.
- Sitemap tells crawlers where your sitemap is.
What robots.txt doesn’t do
This is where most confusion comes from.
robots.txt doesn’t prevent indexing. It controls crawling, not indexing. If other sites link to a page you’ve disallowed, search engines may still index the URL, just without crawling its content. The result can be a search listing with no description. To keep a page out of search results, use a noindex meta tag or header, and make sure the page isn’t blocked in robots.txt, because crawlers need to see the noindex.
robots.txt doesn’t provide security. It’s a public file and a polite request. Malicious bots ignore it. Never use it to hide sensitive areas; listing secret paths in robots.txt actually tells everyone where they are.
Some rules aren’t supported. Google stopped supporting noindex directives in robots.txt in 2019, and it ignores the crawl-delay directive. Other search engines may treat these differently.
WordPress’s virtual robots.txt
WordPress generates a robots.txt file dynamically if no physical file exists. By default, it disallows /wp-admin/ while allowing admin-ajax.php, and since WordPress 5.5, it includes a link to the core sitemap.
SEO plugins often let you edit robots.txt from the dashboard, either by modifying the virtual file or creating a physical one. If a physical robots.txt file exists in your site root, it takes precedence over WordPress’s virtual version, and changes in plugin settings may not apply. Check which version is live by visiting example.com/robots.txt.
The “discourage search engines” setting
Under Settings, then Reading, WordPress has an option called “Discourage search engines from indexing this site”. In modern versions of WordPress, this adds a noindex directive to your pages, rather than relying only on robots.txt.
It’s useful on staging and development sites. It’s catastrophic if left on after launch. I check this setting on every launch, because it’s one of the most common reasons new sites don’t appear in search.
What to disallow
For most WordPress sites, robots.txt should be minimal. Over-blocking causes more problems than under-blocking. Sensible rules include:
- /wp-admin/, with admin-ajax.php allowed, which WordPress does by default.
- Internal search results, like /?s= or /search/, which create endless thin pages.
- Faceted filter and sorting parameters on large stores, if they generate huge numbers of crawlable URLs. Be careful to block only parameters that add no search value.
- Specific crawlers you don’t want, like aggressive SEO tools or AI training crawlers, by user-agent.
What not to block
- CSS and JavaScript files. Search engines render pages like browsers do. Blocking theme or plugin assets in wp-content can prevent Google from understanding your layout and mobile friendliness. Older WordPress advice recommended blocking wp-includes and plugin folders; that advice is outdated and harmful.
- Images, unless you specifically don’t want them in image search.
- Pages you want to remove from search, which need crawlable noindex instead.
- Your whole site, obviously, though “Disallow: /” left over from staging is a surprisingly common mistake.
Testing robots.txt
Search Console includes a robots.txt report showing the version Google fetched and any errors. You can also check specific URLs with the URL Inspection tool to see whether they’re blocked.
After any robots.txt change, confirm the live file at example.com/robots.txt contains what you intended.
Part 2: XML sitemaps
What sitemaps do
An XML sitemap lists URLs on your site that you want search engines to know about, optionally with the date each was last modified. It helps search engines discover pages, especially:
- New pages that don’t have many links yet.
- Large sites where some pages are deep in the structure.
- Sites with weak internal linking.
A sitemap doesn’t guarantee indexing or improve rankings directly. It’s a discovery aid and a statement of which URLs you consider canonical and important.
WordPress core sitemaps
Since version 5.5, WordPress generates basic XML sitemaps automatically at /wp-sitemap.xml. They include posts, pages, public custom post types, taxonomies and author archives.
Core sitemaps are a reasonable baseline, but they’re fairly basic. They don’t automatically exclude noindexed content set by plugins, and they don’t include features like image information.
SEO plugin sitemaps
Most SEO plugins replace core sitemaps with their own, often at /sitemap_index.xml. These typically:
- Exclude content you’ve set to noindex.
- Let you choose which post types and taxonomies to include.
- Include last modified dates and sometimes images.
- Split large sitemaps into multiple files automatically.
If you use an SEO plugin’s sitemap, it usually disables the core sitemap. Make sure only one sitemap system is active, so search engines don’t receive conflicting lists.
What belongs in your sitemap
Include:
- Pages and posts you want indexed.
- Important custom post types, like products, services or case studies.
- Useful taxonomy archives, like main product categories.
Exclude:
- Noindexed pages.
- Redirected URLs.
- Pages returning errors.
- Thin archives, like most tag pages, date archives and author archives on single-author sites.
- Utility pages, like cart, checkout, account, thank-you and login pages.
- Attachment pages, which most sites should redirect or disable.
A clean sitemap sends a clear signal. A sitemap full of redirects, errors and noindexed URLs suggests the site isn’t well maintained and wastes crawl attention.
Last modified dates
The lastmod date tells search engines when a page changed. Google uses it as a hint when it’s consistently accurate. Don’t fake it by updating dates without meaningful changes, as that teaches search engines to ignore your dates.
Large sites
A single sitemap file can contain up to 50,000 URLs and must be under 50 MB uncompressed. Larger sites use a sitemap index that links to multiple sitemap files, which SEO plugins handle automatically.
Submit your sitemap
Submit your sitemap in Google Search Console under Sitemaps, and in Bing Webmaster Tools. Also list it in robots.txt with a Sitemap line, so other search engines can find it.
After submitting, check the report periodically. It shows whether the sitemap was read successfully and how many URLs were discovered. The Page indexing report shows how many sitemap URLs are actually indexed.
Common mistakes
- Leaving “Disallow: /” or the “discourage search engines” setting on after launch.
- Blocking CSS and JavaScript in robots.txt.
- Using robots.txt to try to remove pages from search instead of noindex.
- Having both a physical robots.txt and plugin settings, and editing the wrong one.
- Running two sitemap systems at once.
- Including redirected, noindexed or broken URLs in sitemaps.
- Forgetting to update robots.txt and sitemap URLs after a domain change.
Keep a copy of your working robots.txt in your site documentation, so you can restore it quickly if a plugin or migration changes it.
Frequently asked questions
Do I need a robots.txt file?
WordPress provides a virtual one automatically, which is fine for many sites. Customise it only if you need to block specific paths or crawlers.
Will blocking a page in robots.txt remove it from Google?
Not reliably. It stops crawling, but the URL may still be indexed if linked from elsewhere. Use noindex, and keep the page crawlable so search engines can see it.
Which sitemap should I use, WordPress core or my SEO plugin’s?
Either works. SEO plugin sitemaps usually offer more control and automatically exclude noindexed content. Just make sure only one is active.
How often should I check my sitemap?
After launches, migrations and major structural changes, and periodically through Search Console’s sitemap and indexing reports.
Final thoughts
robots.txt and XML sitemaps are small files with big responsibilities. Keep robots.txt minimal, never block CSS and JavaScript, and remember it controls crawling, not indexing. Keep your sitemap clean, listing only canonical pages you want indexed, with one sitemap system active. Check both after every launch and migration, and use Search Console to confirm search engines see what you intend. Getting these basics right makes everything else in SEO work better.




