Free Robots.txt Tester: Is a URL Blocked?

It looks for the group whose User-agent is the longest prefix of its own name: Googlebot-Image uses the “Googlebot-Image” group if it exists, otherwise “Googlebot”, otherwise “*”. A dedicated group, even an empty one, fully replaces the “*” group — it is not added to it.

The rule with the longest (most specific) pattern. When lengths are equal, Allow wins over Disallow. So “Disallow: /blog” blocks “/blog/article”, unless “Allow: /blog/article” is present.

“*” matches any sequence of characters and “$” marks the end of the URL. “Disallow: /*?sort=” blocks any URL containing that parameter; “Disallow: /*.pdf$” blocks PDFs but not “file.pdf?v=2”.

No: it prevents crawling. A blocked URL can still appear in Google if other pages link to it, but without a snippet. To deindex a page, let the bot access it and use a noindex meta robots tag.

A 404 code is treated as “everything is allowed”. However, repeated server errors (5xx) can lead Google to treat the whole site as disallowed: keep an eye on the file’s availability.

Google ignores them. Bing and Yandex respect Crawl-delay. The Noindex directive in robots.txt has not been supported since 2019: use the meta tag or the X-Robots-Tag header instead.