Mydentify AI Crawler Access Checker
Check whether major AI crawlers can access and index a live URL.
Mydentify AI Crawler Access Checker checks a live URL against documented OpenAI and Anthropic crawler names. It evaluates robots.txt path rules, inspects meta robots and X-Robots-Tag directives, and compares page responses across labeled user-agent requests. I built it to make crawler access problems visible without an account or API key. Its report clearly notes that user-agent probes come from the visitor’s network and cannot prove behavior for official provider IP ranges.
I maintain the project.