Free AI Crawler Access Checker: GPTBot, ClaudeBot

It depends on your goal. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended…) keeps your content out of model training without affecting your visibility. Blocking search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) removes you from the cited answers in ChatGPT Search, Perplexity or Claude — a growing source of traffic. “User” agents only act when a person asks to read your page.

No. Google-Extended only controls whether your content is used to train Gemini. Googlebot still handles search and AI Overviews: blocking it removes you from regular results.

A text file placed at the site root (/llms.txt) that summarizes in Markdown what the site offers and lists its key pages, to help AI assistants understand it. Adoption by model providers is still partial, but it is cheap to set up.

“noai” and “noimageai” are meta robots values proposed by some players to opt out of training; compliance is not guaranteed. “max-snippet” limits the length of the snippet Google can show, including in its AI Overviews.

Yes, technically. The major providers (OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity for its declared crawler) say they respect it. For the rest, only blocking at the server or CDN level (by User-agent or IP range) is enforceable.

robots.txt rules apply per path: “Disallow: /blog/” blocks articles but not the homepage. The tool evaluates the exact URL you enter; test several sections if your rules are granular.