The invisible problem

You ask ChatGPT "where can I buy [your product]?" It recommends three competitors. Your store isn't mentioned. You assume your SEO is bad, your content is weak, or AI just doesn't know you exist.

Often, the real answer is simpler and more frustrating: your site is actively blocking the AI crawlers that would have found you.

This happens in two places — and both are invisible unless you know where to look.

Check your robots.txt

Your robots.txt file tells crawlers what they can and can't access. Many sites — especially those that added "AI protection" during the 2024–2025 panic — have rules like:

User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: /

If you wanted to block AI training, fine. But some of these same crawlers also power AI search — the answers people see when they ask for product recommendations. Blocking the training crawler and blocking the search crawler are different decisions. OpenAI, for example, operates separate crawlers: GPTBot (training), OAI-SearchBot (search), and ChatGPT-User (real-time browsing). Treating them as one "AI bot" is a common, costly mistake.

Check yours: visit yourstore.com/robots.txt and look for Disallow: / under any AI-related user-agent. If you see it and you want AI visibility, that's your first finding.

The CDN/WAF trap

This is the one that catches people who checked robots.txt and found it clean.

Your robots.txt says "allow everything." But your CDN (Cloudflare, Akamai, Fastly), your WAF (web application firewall), or your security plugin is blocking AI crawlers at the network level — through IP restrictions, JavaScript challenges, CAPTCHAs, rate limits, geofencing, user-agent filtering, or automatic bot scoring.

The page loads fine in your browser. The AI crawler gets a 403, a challenge page, or a timeout. You never see it. You just never get cited.

Check yours: look at your CDN or security settings for bot rules. If you see "block AI bots" or "challenge unknown bots" enabled, AI search crawlers are likely being caught. The fix is a narrowly scoped rule that permits legitimate AI search crawlers while keeping your security posture — not disabling security entirely.

Know your bots

The major AI crawlers you should recognize:

  • GPTBot — OpenAI's training crawler. Blocks training data collection.
  • OAI-SearchBot — OpenAI's search crawler. Blocks ChatGPT search citations.
  • ChatGPT-User — OpenAI's real-time browsing. Blocks live "browse the web" answers.
  • PerplexityBot — Perplexity's search crawler.
  • ClaudeBot — Anthropic's crawler for Claude.
  • Google-Extended — Google's AI training control (separate from regular Googlebot).

Each has a different purpose. Blocking GPTBot stops your content from training future models. Blocking OAI-SearchBot stops you from appearing in ChatGPT search answers today. These are not the same decision, and they shouldn't share one rule.

How to test

  1. Read your robots.txt. Look for AI crawler user-agents with Disallow rules.
  2. Check your CDN/security dashboard. Look for bot-blocking rules, especially "AI" or "unknown bot" categories.
  3. Test with a crawler user-agent. Use a tool or ask your developer to fetch your homepage with User-agent: OAI-SearchBot and see if the response matches what a normal browser gets.
  4. Check your server logs. Look for AI crawler user-agents and what status codes they received. 200 = fine. 403 = blocked. 503 = challenged.
  5. Ask an AI. Ask ChatGPT or Perplexity a question your site should answer. If they can't find you and you've ruled out content issues, access is the likely cause.

The GEO audit checks all of this. Every SunmosLab audit includes a per-bot robots.txt review, CDN/WAF warning detection, and a 30-run AI visibility baseline. If your site is blocking AI crawlers, it's a P0 finding — with the exact rule to change and the evidence of what was blocked. Start with a $400 audit or book a free call. If Google itself can't see your store, start with why stores don't show up on Google.