What Is a Noindex Tag?

What Noindex Does

A noindex tag instructs search engines to crawl a specific web page but exclude it from their search index. This means the page content can be accessed by users who navigate directly to the URL, but it will not appear in search results for relevant queries.

Unlike a canonical tag, which is a suggestion to search engines, a noindex tag is a directive that Google reliably respects. When a search engine crawler encounters a noindex tag on a page, it processes the page content and any links on that page, but it will not store the page in its index.

Meta Tag vs X-Robots-Tag Header

There are two primary ways to implement a noindex directive:

1. Meta Tag: This is the most common method and involves adding a <meta> tag within the <head> section of an HTML document.

   <meta name="robots" content="noindex">
   

   This tag tells all search engines not to index the page. To specify for Googlebot only, you would use:

   <meta name="googlebot" content="noindex">
   

2. X-Robots-Tag HTTP Response Header: This method is useful for non-HTML files, such as PDFs, images, or for exercising more fine-grained control at the server level. The X-Robots-Tag is an HTTP header returned with the server's response for a URL.

   For example, an HTTP response header might look like this:

   HTTP/1.1 200 OK
   Date: Tue, 25 Jun 2024 12:00:00 GMT
   X-Robots-Tag: noindex
   Content-Type: text/html; charset=UTF-8
   

   Configuring X-Robots-Tag typically requires server-side modifications, such as in .htaccess files for Apache servers or nginx.conf for Nginx servers.

   For instance, to add an X-Robots-Tag to all PDF files using Apache, you could add this to your .htaccess file:

   <Files ~ "\.pdf$">
     Header set X-Robots-Tag "noindex, nofollow"
   </Files>
   

Noindex vs Robots.txt Disallow

It is common to confuse noindex with the Disallow directive in a robots.txt file. While both relate to controlling search engine behavior, they serve different purposes and have distinct effects.

Feature	Noindex Tag	Robots.txt Disallow
Purpose	Prevents a page from appearing in search results.	Prevents search engine crawlers from accessing a page path.
Effect	Page is crawled but not indexed.	Page is not crawled.
Indexing	Prevents indexing.	Does NOT prevent indexing. A Disallowed page can still be indexed if linked extensively from other sites.
Link Equity	Crawlers can still follow links on a noindexed page.	Link equity is typically not passed from disallowed pages.
Implementation	<meta> tag in HTML or X-Robots-Tag header.	Disallow directive in robots.txt file.

A crucial distinction is that robots.txt Disallow prevents crawling, but it does not prevent indexing. If a page is blocked by robots.txt but heavily linked from other indexed pages, search engines may still infer its content and include it in their index, often showing a "description unavailable" snippet. A noindex tag, conversely, ensures that the page will not appear in search results, while still allowing the page to be crawled and for link equity to flow from it (unless a nofollow directive is also present).

For example, if you have a staging site that you do not want search engines to crawl or index, the best practice is to block it at the server level (e.g., using password protection). If that is not possible, a robots.txt Disallow directive accompanied by a noindex meta tag (if crawlers can access the page despite Disallow) is often used as a fallback, although the noindex tag needs to be accessible to be read. If a page must not be indexed, noindex is the reliable mechanism.

Which Pages Should Be Noindexed

Noindex tags are appropriate for pages that serve a functional purpose for users but offer little or no value to search engine users. Examples include:

• Internal search results pages: These pages often contain duplicate or low-quality content generated dynamically.
• Admin login pages: These are for site administrators and contain no public-facing content.
• "Thank you" pages after form submissions or purchases: These pages confirm user actions but are not intended for general search.
• Pages with duplicate content: If internal pages have highly similar content that cannot be consolidated or canonicalized, noindex can prevent internal competition. For example, old versions of articles or blog posts that are maintained for archival purposes but not meant to rank.
• Staging or development sites: These are non-public versions of a site that should not be indexed.
• Filtered product pages (in e-commerce): While many filtered pages are valuable, some combinations of filters can create thin content. For example, "red size small t-shirts" might be indexable, but "red size small t-shirts for left-handed people" might be too specific if no products exist, resulting in sparse content.
• User profile pages on forums or social sites (if publicly accessible but low-value): If user profiles are generally unoptimized and not meant to attract organic traffic, they can be noindexed.
• Printer-friendly versions of pages: These are duplicates created for a specific user experience.

Common Noindex Mistakes

Incorrect use of noindex can lead to unintended consequences for your site's SEO.

1. Noindexing pages you want to rank: Accidentally applying a noindex tag to important landing pages, product pages, or blog posts will cause them to disappear from search results, leading to a significant drop in organic traffic.
2. Using noindex on pages also blocked by robots.txt: If a page is Disallowed in robots.txt, search engine crawlers cannot access it. If they cannot access it, they cannot read the noindex meta tag within the page's HTML. This means the page might still be indexed if there are enough external links pointing to it. To reliably noindex a page, it must be crawlable.
3. Noindexing pages that pass important link equity: If a noindex page has many internal or external links pointing to it, and those links are valuable, using noindex can effectively stop the flow of that link equity to other pages on your site (unless a nofollow directive is also present, which prevents link equity transfer). While Google states that a noindexed page can still pass PageRank, this occurs only if the page allows crawling and only if the links are not nofollowed.
4. Leaving noindex on after a site migration: If a staging site or development version of a site is deployed to production with a noindex tag still active, the entire site will disappear from search results.
5. Noindexing sitemap.xml: The sitemap file helps search engines discover new and updated pages. Noindexing it prevents it from being processed.

How LemRank Audits Indexation

LemRank's Site Audit tool helps you identify noindex directives and potential indexation issues on your website.

During a site crawl, LemRank will:

• Identify pages with noindex meta tags: These are reported so you can confirm they are intentionally noindexed.
• Detect pages blocked by robots.txt: The audit will show which URLs are disallowed, allowing you to cross-reference with your noindex strategy. You can use the Robots.txt Tester to verify Disallow rules.
• Flag pages that are noindexed and also Disallowed: This helps you identify pages where the noindex directive might not be read by search engines.
• Report non-200 status code pages that are also noindexed: While a noindex tag on a page returning a 4xx or 5xx status code is redundant (as these pages are unlikely to be indexed anyway), LemRank highlights this to ensure clarity in your indexation strategy.

Armed with this information, you can review your website's indexation status, ensure important pages are discoverable by search engines, and prevent low-quality or sensitive content from appearing in search results.

Related

• How to Use the Robots.txt Tester
• What Are Canonical Tags