How to Create and Validate a robots.txt File
A misconfigured robots.txt file can silently block Google from crawling your entire site — and it takes just one misplaced character. This guide walks through creating a correct robots.txt from scratch, explains every directive, and shows you how to validate it before it causes damage.
What robots.txt does and doesn't do
Robots.txt is a plain-text file placed at the root of a domain (e.g., https://example.com/robots.txt) that provides instructions to web crawlers about which pages or sections of the site they are permitted to access. It follows the Robots Exclusion Standard, a convention — not a technical enforcement mechanism — that well-behaved crawlers respect.
The critical distinction: robots.txt controls crawling, not indexing. If you block a URL in robots.txt, Google cannot crawl it. But if other pages link to that URL, Google may still index a shallow entry for it based on link data alone — with no content visible in the snippet. This is the opposite of what most people intend when they "block" a page.
To prevent a page from appearing in search results, you need the noindex directive in the page's HTTP response header or in a meta tag on the page itself. Robots.txt is not the right tool for that job.
Robots.txt also cannot prevent Google from crawling pages that are linked from other sites outside your domain. External backlinks to a blocked page will still result in Google knowing the URL exists. For truly sensitive pages, the correct solution is authentication, not robots.txt.
File syntax and rules
The robots.txt file uses a simple directive-based syntax. Each record (called a "group") begins with one or more User-agent lines specifying which crawler the rules apply to, followed by one or more Disallow or Allow lines. Groups are separated by blank lines.
The file must be in plain text, encoded as UTF-8, and served with a text/plain MIME type. It is case-sensitive on the path portion: Disallow: /Admin and Disallow: /admin are two different rules. The file is served at the exact path /robots.txt — no subdirectory, no filename variation.
Google's robots.txt parser follows RFC 9309, which defines the Robots Exclusion Protocol. This means order matters: more specific rules take precedence over less specific ones. If you have Disallow: / and Allow: /public/, the Allow rule for /public/ wins because it is more specific.
Comments are allowed using a hash character (#). Everything after a # on a line is treated as a comment and ignored by parsers. Comments are useful for documenting why a section is blocked or noting when a rule was added.
Common directives explained
User-agent specifies which crawler a group of rules applies to. "User-agent: *" applies to all crawlers. "User-agent: Googlebot" applies only to Google's main web crawler. "User-agent: Googlebot-Image" targets only Google's image crawler. You can target multiple crawlers with multiple User-agent lines before the Disallow/Allow lines.
Disallow specifies a path that the crawler should not access. "Disallow: /admin/" blocks everything under the /admin/ directory. "Disallow: /" blocks the entire site. "Disallow:" (empty value) explicitly allows everything — it is the default, but some webmasters include it for clarity.
Allow specifies a path that the crawler may access, overriding a broader Disallow rule. Allow is most useful to carve exceptions: block /wp-admin/ but allow /wp-admin/admin-ajax.php, which is needed for some WordPress front-end functionality.
Sitemap provides the absolute URL of your XML sitemap. Google uses this to efficiently discover pages. You can include multiple Sitemap directives. Unlike other directives, Sitemap does not belong to a group — place it outside any User-agent group, typically at the end of the file.
Crawl-delay is supported by some crawlers (Bing, Yandex) but not by Google, which controls crawl rate through Google Search Console instead. "Crawl-delay: 10" tells supported crawlers to wait 10 seconds between requests. Do not rely on this for Google.
Common robots.txt mistakes that hurt SEO
Blocking CSS and JavaScript is one of the most damaging mistakes. Google renders pages like a browser and needs access to CSS and JavaScript to understand your page's layout and dynamic content. If you have "Disallow: /wp-content/" and your theme or scripts live in that directory, Google sees only the raw HTML without styling — often misclassifying the page's structure as a result.
Blocking crawl of your entire site during development and forgetting to remove the block before launch is a classic catastrophic error. The rule "Disallow: /" will prevent all crawling, and if it ships to production, your site falls out of the index within weeks. Always audit your production robots.txt file on launch day.
Inconsistent trailing slashes create ambiguity. "Disallow: /admin" blocks /admin but may not block /admin/ (with a trailing slash) in all parsers. Use "Disallow: /admin/" consistently to cover the directory, and if you need to block both the directory and any file at that exact path, add both rules.
Blocking the sitemap URL itself: "Disallow: /sitemap.xml" tells crawlers not to access the file that helps them efficiently discover your pages. This is self-defeating.
Relying on robots.txt to protect sensitive data is a security mistake. Robots.txt is publicly accessible and tells malicious crawlers exactly which directories contain sensitive content. Authentication is the only real protection.
How to test your robots.txt
Google Search Console provides a robots.txt tester (under Settings) that shows exactly how Googlebot reads your file, highlights syntax errors, and lets you test individual URLs to see whether they are allowed or blocked. This is the authoritative tool because it shows Google's actual parsing of your file.
SerpGem's Robots.txt Tester lets you paste any robots.txt content and test whether a specific URL pattern would be allowed or disallowed under those rules. Use this to validate your file before deploying it or to audit a competitor's crawler rules.
The Robots.txt Validator tool checks the syntax of your file — verifying that all directives are correctly formatted, that User-agent groups are properly structured, and that there are no common errors like incorrect path formats or missing blank lines between groups.
After deploying a new robots.txt, monitor Google Search Console for crawl coverage changes over the following 2 to 4 weeks. A sharp drop in crawled pages indicates a rule may be blocking more than intended. Use the URL Inspection tool to check specific pages and see whether Googlebot is allowed to crawl them.
Tools mentioned in this guide
Frequently asked questions
- Does robots.txt prevent pages from appearing in Google's index?
- No. Robots.txt only controls crawling. If other sites link to a blocked page, Google may still index a shallow entry for it based on those links, even without crawling the content. To prevent indexing, use the noindex directive in the HTTP headers or in a meta robots tag on the page.
- What happens if I have no robots.txt file?
- Search engines default to allowing crawling of all public content. A missing robots.txt is not an error — Google will crawl your site as if all paths are allowed. You only need a robots.txt if you want to restrict crawlers from certain sections.
- How quickly does Google pick up changes to robots.txt?
- Google re-fetches robots.txt periodically — typically every 24 hours for active sites, though it can take up to 7 days for the changes to be fully reflected in crawling behavior. Use Google Search Console's robots.txt tester to verify the current version Google sees.
- Can I use robots.txt to block specific file types like PDFs?
- Yes. "Disallow: /*.pdf$" (using the wildcard and end-of-pattern anchor) will block all PDFs from crawling in parsers that support wildcards, including Googlebot. Verify this using a robots.txt testing tool since wildcard support varies by crawler.