Robots Exclusion Checker

The Robots Exclusion Checker is a browser extension designed for SEO professionals and web developers who need to quickly identify any robots exclusion directives that might prevent a page from being crawled or indexed by search engines. It provides a visual dashboard that reports on six key elements affecting search engine access: robots.txt rules, meta robots tags, AI bot exclusions, X-Robots-Tag headers, canonical tags, and rel attribute values such as nofollow, UGC, and sponsored. The extension continuously monitors the current page and updates its status as you navigate, making it a practical tool for auditing complex sites, reviewing faceted navigation, or debugging indexation issues during development.

Key Features

• Robots.txt Rule Detection: When a page is affected by an Allow or Disallow directive in the robots.txt file, the extension displays the specific rule and highlights it within the full robots.txt content. This helps you quickly copy the rule or visit the live file. The feature is especially useful for sites with large or complex robots.txt files where manual inspection is error-prone.
• Meta Robots Tag Analysis: The extension scans the HTML source for meta robots directives like index, noindex, follow, and nofollow, and assigns color-coded icons (green, amber, red) based on their impact on indexation. Non-blocking directives such as nosnippet or noodp are shown but do not trigger alerts. This allows you to see all directives at a glance and verify correct implementation.
• AI Bot Exclusion Monitoring: The extension checks if the site's robots.txt blocks AI companies from accessing content. It tracks 14 bots from six companies (OpenAI, Anthropic, Google, Perplexity, Meta, Apple) across three access types: training data collection, search indexing, and real-time browsing. When exclusions are detected, an "AI" label appears over the extension icon. This feature can be disabled in settings if not needed.
• X-Robots-Tag Header Inspection: The extension parses HTTP response headers for X-Robots-Tag directives and highlights any exclusions. It also displays the full HTTP header with the relevant directives emphasized, making it easy to spot directives that might otherwise be hidden in the response.
• Canonical Tag Monitoring: Although canonical tags do not directly affect indexation, they influence how URLs appear in search results. The extension checks both the HTML and HTTP header for canonical tags. If the current URL differs from the canonical URL, an amber icon alerts you to a potential mismatch. This is particularly useful for sites with duplicate content or URL parameters.
• Nofollow, UGC, and Sponsored Link Highlighting: The extension can visually highlight visible links that use rel="nofollow", "ugc", or "sponsored" attributes. You can choose which types to highlight and set custom colors for each. This feature is optional and can be disabled entirely. It helps in quickly assessing link equity distribution on a page.
• User-Agent Simulation: In the settings, you can select from four user-agents (Googlebot, Googlebot News, Bing, Yahoo) to simulate how each search engine sees the page. This allows you to test if specific bots are blocked differently.
• Multi-Language UI: The extension interface is available in English, German, and Spanish, making it accessible to a broader audience.
• Advanced Navigation Handling: The extension supports JavaScript pushState and back-forward navigation, ensuring it works correctly on single-page applications (SPAs) and sites that update content dynamically without full page reloads.
• Site Exclusion List: You can add domains to an exclusion list to skip checking for those sites, which is helpful when you want to avoid processing on irrelevant or internal domains.

This extension is designed to replace multiple separate tools for checking nofollow links, meta tags, and robots.txt. By consolidating these checks into one interface, it reduces browser overhead and streamlines the audit process. It is suitable for daily use by SEO specialists, content managers, and developers who need to ensure their pages are accessible to search engines and AI crawlers.