Robots.txt
A plain-text file at the root of a website that tells crawlers which URLs they are allowed or disallowed from accessing.
Definition
The robots.txt file is a plain text file placed at the root of a domain (e.g., https://example.com/robots.txt) that communicates crawl directives to web robots through the Robots Exclusion Protocol (REP). It instructs compliant bots (Googlebot, Bingbot, etc.) which sections of the site they are allowed or not allowed to request. The file uses User-agent directives to target specific bots and Allow/Disallow directives to specify URL patterns.
Robots.txt is not a security mechanism — it is a polite request that well-behaved bots honor but malicious scrapers ignore. Its primary SEO use is crawl budget management: by disallowing low-value URLs (admin pages, search result pages, faceted navigation parameters, staging areas), you conserve crawl budget for content you want indexed.
Critical mistakes include accidentally blocking CSS and JavaScript files (which prevents Google from rendering pages correctly), blocking entire site sections that should be indexed, and creating overly permissive rules that expose private areas. The robots.txt syntax requires precise formatting; a single misspelling in a directory path will negate the rule. Use a robots.txt validator and tester to verify your rules work as intended before deploying changes that could accidentally block critical content from crawling.