How the tester works
Fetch a site’s robots.txt or paste your own, choose a user-agent and enter a path. The tester applies the same matching rules Google documents: the most specific (longest) matching rule wins, and Allow wins a tie. Wildcards (*) and end anchors ($) are supported.
This is an educational tool — always confirm critical changes with Google Search Console.
Frequently asked questions
Does Disallow remove a page from Google?
No. Disallow blocks crawling, not indexing. To keep a page out of results, allow crawling and use a noindex tag.
Can I block AI crawlers?
Many AI crawlers publish their user-agent (for example GPTBot or ClaudeBot) and respect robots.txt rules aimed at them.