← All tools / AI crawler rules

Digital Braid / Free utility

What do your crawler rules actually say?

Enter a website or paste robots.txt. Test a page path against six documented crawlers and see the exact rule behind each result.

Choose your input

Fetches the public /robots.txt file. The path above is evaluated against its rules; that page is not crawled.

What this checks

Standard Allow and Disallow rules, wildcard paths, end-of-path matches, and combined groups for the same crawler. A crawler’s own group takes precedence over the wildcard group. More specific rules win; Allow wins a tie.

This is a rules diagnostic. It does not test firewall blocks, JavaScript rendering, HTTP headers, indexing, or citations. User-triggered fetchers and Google-Extended policy controls are not included.