robots.txt Validator
Check if your site is accessible to Google, Bing, and the top AI crawlers — no account required.
No data is stored or shared.
Understand robots.txt
What is robots.txt and why it matters
The robots.txt file lives at the root of your site and tells web crawlers which pages they can or cannot access. A misconfigured file can silently block Google — and no indexed pages means no visits.
With AI assistants like ChatGPT and Perplexity growing, robots.txt now also controls which language models can learn from your content. Knowing what's blocked is the first step to appearing in AI-generated answers.
This validator only reads the public robots.txt file of the domain. It does not access password-protected pages or private data.
Why validate
Three good reasons to check now
Guaranteed indexing
An accidental Disallow: / blocks your entire site. Find and fix it before traffic drops.
AI visibility
GPTBot, ClaudeBot, and PerplexityBot respect robots.txt. Blocking them removes your content from AI assistant answers.
Sitemap confirmed
Declaring your Sitemap in robots.txt speeds up new page discovery by Google and Bing.
How it works
Results in three steps
Enter your domain
Type your site address. No account needed.
Automatic analysis
We fetch your robots.txt and check syntax, sitemap, and access for each crawler.
Fix with confidence
Get a clear diagnosis with each crawler's status and improvement guidance.
Reference
AI crawler categories
| Category | Crawlers | Control | Impact |
|---|---|---|---|
| Search | Googlebot, Bingbot | Disallow affects ranking | Critical |
| AI search | OAI-SearchBot, PerplexityBot | Disallow removes from AI answers | High |
| Assistant | ChatGPT-User, ClaudeBot | Disallow removes from browsing mode | High |
| Training | GPTBot, Google-Extended, CCBot | Disallow affects future models | Medium |
Frequently asked questions
robots.txt questions
What is a robots.txt file?
It is a public text file at the root of your domain (at /robots.txt) that follows the Robots Exclusion Protocol (RFC 9309). It tells search and AI crawlers which paths they may or may not crawl. It is not a security mechanism — it only guides crawlers that respect the standard.
Does robots.txt stop a page from appearing on Google?
No. The Disallow directive blocks crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To remove it from search, use the robots meta tag noindex or the X-Robots-Tag header — and the page must not be blocked in robots.txt, otherwise Google never sees the noindex.
How do I allow or block AI crawlers like GPTBot and Google-Extended?
Create a group with a specific User-agent followed by the rules. For example, User-agent: GPTBot with Disallow: / blocks the whole site for OpenAI's bot. Google-Extended controls whether your content is used for AI training and Gemini without affecting search Googlebot. Blocking these agents lowers your chances of being cited by ChatGPT, Perplexity and Gemini.
Is robots.txt case-sensitive?
The file name must be exactly robots.txt in lowercase. Directive names are not case-sensitive, but paths are: /Folder/ and /folder/ are treated as different. It is worth double-checking every rule.
Do I need to declare my sitemap in robots.txt?
It is not required, but it is recommended. The Sitemap directive, with the absolute URL of your sitemap.xml, helps search engines discover all your URLs faster. It can appear on any line of the file, regardless of the User-agent groups.
Why does the tool show AI crawlers as allowed or blocked?
We list the main search and AI crawlers so you can see at a glance who can reach your content. In GEO (generative engine optimization), allowing AI crawlers is a prerequisite for your site to be read and cited in answers generated by assistants.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI's training crawler — it learns from your content to improve future models. ChatGPT-User is ChatGPT's browsing mode — it reads pages in real time to answer user questions.
Does blocking Google-Extended affect Googlebot?
No. Google-Extended only controls use of your content in Gemini and Google AI training. Search Googlebot continues to work normally even with Google-Extended blocked.
My site is new. Do I need robots.txt?
Yes. Even new sites should have a basic robots.txt with User-agent: *, Allow: / and a sitemap reference. This ensures search engines and AI crawlers can find and index your content from the start.