How to Use the Robots.txt Tester

What robots.txt does

The robots.txt file is a plain text file hosted at the root of a website (for example, https://www.example.com/robots.txt). It provides instructions to web crawlers, such as Googlebot, about which parts of a site they are permitted or not permitted to access. This file is part of the Robots Exclusion Protocol (REP).

The primary function of robots.txt is to manage crawler access, not to control indexing. If a page is disallowed in robots.txt, search engines generally will not crawl it. However, if other pages link to the disallowed page, or if the page is listed in a sitemap, search engines might still discover and index its URL, even if they cannot access its content. To prevent indexing, use noindex meta tags or HTTP headers.

Common uses for robots.txt include:

• Preventing the crawling of duplicate content.
• Keeping sections of a website (e.g., staging environments, admin pages) private from search engines.
• Managing crawl budget by preventing crawlers from accessing unimportant pages.

How the tester works

The LemRank robots.txt tester helps determine whether a specific URL on your website is blocked by your robots.txt file for a given user-agent.

To use the tester:

1. Enter the full URL you want to test in the "URL" field. For example, https://www.example.com/category/product-page.html.
2. Select the user-agent you want to simulate in the "User-agent" dropdown. Options include common crawlers like Googlebot, Bingbot, and AdsBot, as well as a generic * which represents all crawlers not specifically named.
3. Click "Test URL".

LemRank fetches the robots.txt file from the root domain of the URL you provided. For example, if you input https://www.example.com/blog/article-1, the tester will look for the robots.txt file at https://www.example.com/robots.txt. The tool then parses this file using the same precedence rules documented by Google for its crawlers.

The robots.txt tester does not crawl the URL or check its live status. It only evaluates the instructions within your robots.txt file against the provided URL and user-agent.

Interpreting "allowed" vs "blocked" results

After you run a test, the tool will display one of two primary results:

• Allowed: This indicates that, according to your robots.txt file and the specified user-agent, the URL is permitted to be crawled. This does not guarantee that the page will be crawled or indexed, only that robots.txt is not preventing it.
• Blocked: This indicates that, according to your robots.txt file and the specified user-agent, the URL is not permitted to be crawled. Search engines that respect robots.txt will generally refrain from accessing this URL's content.

Alongside the result, the tester will highlight the specific Disallow or Allow directive within your robots.txt file that caused the result, providing context for the decision.

For example, if your robots.txt contains:

User-agent: Googlebot
Disallow: /private/

And you test https://www.example.com/private/document.html with Googlebot, the result will be "Blocked" and the line Disallow: /private/ will be highlighted.

If you test https://www.example.com/public/page.html with Googlebot, the result will be "Allowed" because no rule explicitly blocks it.

Common robots.txt mistakes that block Google

Misconfigurations in robots.txt can inadvertently prevent search engines from accessing important parts of your website. Here are some common mistakes:

• Accidental blanket disallow: One of the most frequent errors is an overly broad Disallow directive. For example, Disallow: / effectively blocks all user-agents from crawling the entire website. This often happens temporarily during development or staging and is forgotten when the site goes live.
• Blocking CSS/JS files: While it might seem intuitive to block directories containing stylesheets (.css) or JavaScript (.js) files to optimize crawl budget, doing so can prevent search engines from rendering your pages correctly. This can lead to indexing issues because Googlebot needs to understand the page's layout and interactivity to properly evaluate its content and user experience.
• Syntax errors: Minor typos or incorrect formatting can render robots.txt directives ineffective or cause unintended blocks. For example, a missing colon or an incorrect directive name can be problematic.
• Blocking sitemaps: If your robots.txt file accidentally blocks the path to your sitemap (e.g., Disallow: /sitemap.xml), search engines may have difficulty discovering all the URLs on your site. The sitemap path should generally be explicitly allowed or at least not disallowed.
• Conflicting directives: When multiple directives apply to a URL, understanding precedence is crucial. Sometimes, a more specific Disallow can override a general Allow, or vice-versa, leading to unexpected blocks.

Regularly testing important URLs with the robots.txt tester can help identify and rectify these issues before they negatively impact your search visibility.

Wildcards and precedence rules

The robots.txt file supports wildcards to make directives more flexible.

• The asterisk * matches any sequence of characters.
• The dollar sign $ designates the end of a URL.

Examples:

• Disallow: /*/private/ blocks any URL with /private/ in its path, such as /folder/private/page.html or /another/private/section/.
• Disallow: /images/*.jpg$ blocks all .jpg files within the /images/ directory.

When multiple rules apply to a URL, specific precedence rules determine the outcome:

1. Longest match wins: Between conflicting Allow and Disallow rules for the same user-agent, the directive with the most characters in its path normally takes precedence. For example, if a robots.txt contains Disallow: /a and Allow: /a/b, then Allow: /a/b applies to /a/b/c.
2. Order of Disallow vs. Allow: If rules are identical in length, Disallow generally takes precedence over Allow for Googlebot. For other crawlers, the outcome might depend on their specific implementation or the order of rules. However, Google's documentation states that for Googlebot, a more specific Allow rule will override a less specific Disallow rule.

Consider this example for Googlebot:

User-agent: Googlebot
Disallow: /folder/
Allow: /folder/specific-page.html

If you test /folder/another-page.html, it would be blocked by Disallow: /folder/. If you test /folder/specific-page.html, it would be allowed by Allow: /folder/specific-page.html because the Allow rule is more specific.

The LemRank robots.txt tester evaluates these rules based on Google's documented precedence, providing an accurate assessment for Googlebot and other common crawlers.

Fixing a blocked page

If the robots.txt tester indicates that an important page on your site is blocked, here are steps to fix it:

1. Identify the blocking directive: The tester will show you the specific Disallow rule causing the block. Review this rule to understand why it was put in place.
2. Edit your robots.txt file: Access your website's robots.txt file via FTP, your hosting control panel, or your content management system's file manager.
3. Modify or remove the directive:
   • Remove a blanket disallow: If your robots.txt contains Disallow: / and it's blocking your entire site, remove this line or replace it with specific Disallow directives for sections you truly want to block.
   • Add an Allow directive: If a general Disallow rule is blocking a specific important page, you can add a more specific Allow directive. For example, if Disallow: /blog/ is present, but /blog/important-article.html needs to be crawled, add Allow: /blog/important-article.html.
   • Refine pathways: Adjust the paths in your Disallow rules to be more precise, ensuring they only block the intended sections.
4. Save and upload: Save the modified robots.txt file and upload it to the root directory of your website, replacing the old file.
5. Re-test with the LemRank tester: Use the robots.txt tester again with the same URL and user-agent to confirm that the changes have taken effect and the page is now "Allowed".
6. Request re-crawling (optional): For Google, you can use the URL Inspection tool in Google Search Console to request re-crawling of the specific URL once the robots.txt block is resolved. This helps expedite the discovery of your changes.

Keep in mind that changes to robots.txt can take some time for crawlers to discover and implement, depending on their crawl schedule.

Related

• How to Use the Sitemap Validator