SerpGem
On-Page SEO

Meta Robots Tag

An HTML directive that controls whether search engines can index a page and follow its links.

Definition

The meta robots tag is an HTML <meta> element placed in the <head> section that gives page-level instructions to search engine crawlers. Its most common values are index/noindex (whether the page should appear in search results) and follow/nofollow (whether the crawler should follow links on the page). A page with <meta name="robots" content="noindex, nofollow"> tells all crawlers not to index it or follow any links on it.

Unlike the robots.txt file (which controls crawl access), the meta robots tag works at the indexing level. A crawler must be able to reach the page to read the meta robots tag, so blocking a page in robots.txt and also adding noindex creates a contradiction — Google can't see the noindex directive if it can't crawl the page. This is a common technical SEO mistake.

Additional directives include: nosnippet (prevent text/image snippets in results), max-snippet:N (limit snippet character length), max-image-preview:none/standard/large, and max-video-preview:N. These are especially relevant since Google may generate snippets from third-party content syndicated across the web. The X-Robots-Tag HTTP response header achieves the same effect for non-HTML files (PDFs, images) where placing a meta tag is not possible.

Frequently Asked Questions

What's the difference between robots.txt and the meta robots tag?
robots.txt controls crawl access — whether a bot is allowed to visit a URL. The meta robots tag controls indexing — whether a visited page should appear in search results. You need to allow crawling for the noindex directive to take effect, because Google must crawl the page to read the tag.
If I add noindex to a page, will it be removed from Google immediately?
No. After adding noindex, you must wait for Google to recrawl the page and then process the de-indexing signal. This can take days to weeks depending on crawl frequency. You can request an indexing check via Google Search Console to speed up crawl, but removal from the index is not instant.