CrawlerCheck
Instantly check if Googlebot, GPTBot, ClaudeBot, Bingbot, FacebookBot and 100+ other crawlers are allowed or blocked by your robots.txt, meta tags and headers.
We fetch your robots.txt first. If it disallows the crawler's user-agent, the path is blocked before anything else is checked.
If allowed by robots.txt, we then check meta robots tags and the X-Robots-Tag HTTP header for indexability rules.
No signup, no API key. One URL → full category-by-category report in seconds, cached for fast re-runs.
How Crawler Check Works
Crawler Check analyses the three places that decide whether a bot can see your content — robots.txt, meta robots tags and X-Robots-Tag HTTP headers. Each crawler in our database is checked against these sources so you know exactly what is allowed, what is blocked, and where to fix it.
Check robots.txt permissions
Enter any URL and we read its robots.txt, resolve every user-agent rule, and report whether each crawler may crawl the page.
Inspect hidden headers
A plugin or server can add a hidden X-Robots-Tag: noindex you can't see in HTML. We surface it instantly.
Validate AI bot access
Confirm ChatGPT, Claude, Gemini and Perplexity bots are not accidentally blocked — or correctly blocked if you want.
Save crawl budget
Block low-value bots so Google focuses crawl budget on the pages that matter most.
Supported crawlers
Crawler Check covers 100+ user-agents across 8 categories — every major search engine, AI/LLM crawler, social sharing bot, SEO tool, scraper, cloud service and Google-specific bot. Click any category above to see the full list and its allow/block status.
Frequently asked questions
How can I check if Googlebot is blocked?
Enter your URL above — we fetch your robots.txt and resolve the rule for Googlebot specifically. If blocked, the report shows the exact Disallow pattern and where to fix it.
Do AI crawlers like GPTBot or ClaudeBot respect robots.txt?
Most do, but some (notably ChatGPT-User) are user-initiated and may ignore robots.txt. Our report flags this so you know which rules are actually enforceable.
Can I test a specific page, not just the homepage?
Yes — enter the full URL including the path (e.g. https://example.com/blog/post). We check robots.txt rules for that exact path.
Why would a bot be listed as Allowed even though I added noindex?
robots.txt controls crawling, not indexing. A noindex meta tag or X-Robots-Tag header stops indexing but allows the crawl — our Gate 2 analysis covers this.
Is this tool free?
Yes — completely free, no signup, no API key required. Just enter a URL and see your results.
More tools for ChatGPT Ads
Try the other free tools across the OpenAI Ads ecosystem.
Ridoway Pixel
Free WordPress plugin for OpenAI Ads conversion tracking — browser pixel + Conversions API, 5-minute setup.
ChatGPT Pixel Helper
Chrome extension that inspects oaiq calls, beacons, cookies, and CAPI misuse in real time.
Context Hint Generator
Turn product, offer, and audience into ready-to-paste context hints in your browser — free.
Context Hint Plugin
Research buyer conversations, nearby advertisers, and validated hints inside Claude, ChatGPT, or Codex.
Reach Calculator
Estimate audience reach and impressions across ChatGPT Ads placements and countries.
CPC Budget Calculator
Turn a monthly budget into expected clicks, impressions, and conversions for ChatGPT Ads.