Where does robots.txt need to be located?

robots.txt must be at the root of your domain: https://example.com/robots.txt. It cannot be in a subdirectory. If you have multiple subdomains, each needs its own robots.txt: https://blog.example.com/robots.txt controls the blog subdomain independently of the main domain.

Does robots.txt affect Google rankings?

Indirectly, yes. Blocking crawlers from important pages prevents those pages from being indexed and ranking. Blocking CSS and JavaScript can prevent Google from rendering your pages correctly, reducing their ability to assess content quality. Blocking low-value pages (admin, staging) can improve crawl budget allocation to important pages.

Can robots.txt prevent a page from appearing in search results?

No. A page blocked by robots.txt can still appear in search results if another site links to it — Google may index the URL without crawling the page content. To prevent a page from appearing in search results at all, use a "noindex" meta tag or X-Robots-Tag response header instead.

What is the maximum size for a robots.txt file?

Google processes up to 500 kilobytes (500KB) of robots.txt content. Rules beyond 500KB are ignored. For extremely large sites with complex crawl rules, consider whether the rules can be simplified or whether subdomain-specific robots.txt files would be more maintainable.

Technical SEO Tool

Robots.txt Validator

Paste your robots.txt content or fetch it from a URL. Validates syntax, parses crawl rules, and warns you if Googlebot or Bingbot are accidentally blocked.

InputFetch from URL· Enter a domain to fetch its robots.txt

InputPaste robots.txt content· Or paste content directly

OutputValidation results· Awaiting validation

Paste robots.txt content and click Validate, or fetch from a domain above.

Robots.txt Guide

How robots.txt works and why mistakes are critical SEO errors

Robots.txt errors are among the most impactful — and most common — technical SEO mistakes. A single misplaced Disallow: / can block Googlebot from indexing your entire site within hours. Understanding robots.txt syntax, directives, and best practices protects you from catastrophic, traffic-destroying misconfigurations.

robots.txt is a suggestion, not a gate

Robots.txt is a voluntary standard — it relies on crawlers obeying the Robots Exclusion Protocol. Well-behaved bots (Googlebot, Bingbot, DuckDuckBot) respect it. Malicious scrapers, vulnerability scanners, and spam bots ignore it completely. robots.txt protects your crawl budget and indexing, not your content from bad actors.

User-agent: the bot identifier

Each block in robots.txt starts with one or more User-agent: lines. "User-agent: *" applies to all crawlers. "User-agent: Googlebot" applies only to Google's main crawler. User-agent names are case-insensitive. A block applies until the next User-agent: directive or end of file.

Disallow: — what not to crawl

Disallow: / blocks everything. Disallow: /private/ blocks that path and everything under it. Disallow: with an empty value allows everything (it is equivalent to no rule). The order of Allow: and Disallow: rules matters — the most specific matching rule wins. Google uses the longest matching rule.

Allow: — override a Disallow

Allow: lets you carve exceptions out of a Disallow. Example: "Disallow: /products/" followed by "Allow: /products/featured/" tells the crawler to block all /products/ except /products/featured/. Allow: is not in the original RFC but is supported by all major crawlers.

Sitemap: directive

The Sitemap: directive in robots.txt tells crawlers where to find your XML sitemap. You can have multiple Sitemap: lines. This is a reliable way to communicate your sitemap to all crawlers, not just Googlebot via Search Console. Format: "Sitemap: https://example.com/sitemap.xml"

Crawl-delay: use with caution

Crawl-delay: specifies seconds between crawler requests. Googlebot ignores Crawl-delay — use Google Search Console's crawl rate setting instead. Bingbot and many other crawlers respect it. A Crawl-delay of 10 (10 seconds) will drastically slow crawling of large sites. Only use if server load is a genuine concern.

Pro Tips

Test before deploying

Always validate your robots.txt before deploying. Use this tool on your draft, then confirm with Google Search Console's robots.txt tester after deploying. A broken robots.txt deployed to production can deindex pages within 24-48 hours.

Disallow: / is the nuclear option

"Disallow: /" under "User-agent: *" blocks all crawlers from your entire site. This is the most common catastrophic robots.txt error. It is sometimes left in accidentally after development (when blocking all crawlers during dev is intentional). Always check for this pattern.

Comments start with #

Use # comments to document your robots.txt rules. "# Block crawling of admin area" before the relevant Disallow makes your file self-documenting and reduces the chance of future editors accidentally removing a rule they don't understand. Google and all major crawlers ignore comment lines.

Frequently Asked Questions

Where does robots.txt need to be located?: robots.txt must be at the root of your domain: https://example.com/robots.txt. It cannot be in a subdirectory. If you have multiple subdomains, each needs its own robots.txt: https://blog.example.com/robots.txt controls the blog subdomain independently of the main domain.
Does robots.txt affect Google rankings?: Indirectly, yes. Blocking crawlers from important pages prevents those pages from being indexed and ranking. Blocking CSS and JavaScript can prevent Google from rendering your pages correctly, reducing their ability to assess content quality. Blocking low-value pages (admin, staging) can improve crawl budget allocation to important pages.
Can robots.txt prevent a page from appearing in search results?: No. A page blocked by robots.txt can still appear in search results if another site links to it — Google may index the URL without crawling the page content. To prevent a page from appearing in search results at all, use a "noindex" meta tag or X-Robots-Tag response header instead.
What is the maximum size for a robots.txt file?: Google processes up to 500 kilobytes (500KB) of robots.txt content. Rules beyond 500KB are ignored. For extremely large sites with complex crawl rules, consider whether the rules can be simplified or whether subdomain-specific robots.txt files would be more maintainable.