Robots.txt Explained: Control Search Engine Crawling

A robots.txt file is a simple text file at the root of your website that tells search engine crawlers which pages to visit and which to skip. It is one of the oldest and most basic SEO tools, and when used correctly, it helps you manage how search engines interact with your site.

This guide covers what robots.txt is, how the directives work, common mistakes, and how to test your configuration.

What is robots.txt?

When a search engine crawler like Googlebot visits your site, the first thing it looks for is https://yourdomain.com/robots.txt. If the file exists, the crawler reads its instructions before visiting any other page.

A basic robots.txt file looks like this:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

The User-agent: * line means "apply these rules to all crawlers." Allow: / says "you may crawl everything." The Sitemap directive tells the crawler where to find your sitemap.

How the directives work

Robots.txt uses a small set of directives. Understanding these is enough for most situations:

User-agent

Specifies which crawler the rules apply to. * means all crawlers. You can also target specific bots:

User-agent: Googlebot
Disallow: /private/

This tells Googlebot specifically to skip the /private/ directory, while other crawlers follow the general rules.

Disallow

Tells the crawler not to visit the specified path:

User-agent: *
Disallow: /admin/
Disallow: /search?q=

The path is relative to your domain root. /admin/ blocks https://example.com/admin/ and everything under it.

Allow

Override Disallow for a specific path. This is useful when you want to block a directory but allow one page inside it:

User-agent: *
Disallow: /tools/
Allow: /tools/image-compressor

Google processes directives top to bottom and uses the most specific match. In this example, /tools/ is blocked except for the image compressor page.

Sitemap

Points the crawler to your sitemap. This is not a crawl directive but a helpful signal:

Sitemap: https://example.com/sitemap.xml

You can include multiple sitemap directives if you have several sitemaps.

Practical robots.txt examples

Here are configurations for common situations:

Allow everything (default for most sites):

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Block all crawlers from the entire site:

User-agent: *
Disallow: /

Block a specific directory:

User-agent: *
Disallow: /internal/
Disallow: /tmp/
Allow: /

Block one crawler but allow others:

User-agent: BadBot
Disallow: /

User-agent: *
Allow: /

Allow Google but block Bing:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Disallow: /

What robots.txt can and cannot do

This is the most important part of understanding robots.txt:

Robots.txt CAN:

• Prevent crawlers from visiting specific URLs or directories
• Guide crawlers toward the most important pages
• Save crawl budget on large sites by blocking low-value pages
• Point crawlers to your sitemap

Robots.txt CANNOT:

• Guarantee a page will not appear in search results
• Hide URLs from Google entirely (the URL can still appear if linked from another page)
• Block all crawlers (malicious bots ignore robots.txt)
• Replace a noindex tag for preventing indexing

The difference between crawling and indexing

These terms are related but distinct:

• Crawling is when a search engine bot visits a page and reads its content
• Indexing is when Google stores that content in its database and may show it in search results

Robots.txt controls crawling. A page blocked by robots.txt will not be crawled, but if the URL is linked from elsewhere, Google may still list it in search results (though without the content, since it could not read the page).

This is why robots.txt alone is not a privacy tool. For pages that should never appear in search, combine robots.txt with noindex tags or authentication.

Common robots.txt mistakes

These mistakes happen frequently and can hurt your SEO:

1. Blocking CSS and JavaScript - Google needs these to render your page. Blocking them makes your pages look broken in search results.
2. Blocking the entire site by accident - a Disallow: / without a more specific User-agent blocks everything.
3. Forgetting to update after site changes - new sections may need new rules.
4. Using robots.txt instead of noindex - robots.txt prevents crawling but not indexing. Use noindex when you want to remove a page from search results.
5. Blocking the sitemap - if your robots.txt blocks the sitemap URL, crawlers cannot find it.
6. Not including a Sitemap directive - while not required, it helps crawlers find your sitemap quickly.

How to test your robots.txt

Testing is straightforward:

1. Visit https://yourdomain.com/robots.txt in a browser to see the current file
2. Use Google Search Console's robots.txt Tester (under Settings > Crawling) to test specific URLs
3. Check the Coverage report to see if important pages are being blocked
4. After making changes, use the URL Inspection tool to verify Google can still crawl key pages

Google caches your robots.txt, so changes may take a day or two to take effect.

The generator produces a clean robots.txt file with the correct syntax, so you avoid the formatting errors that can accidentally block crawlers.

Connecting robots.txt to the rest of SEO

Robots.txt is part of your site's crawl management. It works alongside XML sitemaps (which tell crawlers what exists), meta tags (which tell crawlers what pages are about), and Open Graph tags (which control social sharing). Together, these elements form the technical foundation of your site's search presence.

SEO tools that help

Robots.txt Generator

SEO Tools

Generate robots.txt files to control search engine crawling. Configure allow/disallow rules and crawl delays.

Sitemap Generator

SEO Tools

Generate XML sitemaps for your website. Add URLs with priority, change frequency, and last modified date.

Meta Tag Generator

SEO Tools

Generate SEO meta tags for your website including title, description, keywords, Open Graph, and Twitter Card tags.