Your company ranks on page one of Google. Your reviews are solid. Your content is thorough. But when you ask ChatGPT to recommend businesses in your category, your name doesn’t appear. Competitors do. That’s the specific frustration that brought us to investigate our own AI visibility at CROSS Digital, and what we found surprised us.
The short version: our Web Application Firewall was silently blocking AI crawlers. No error page. No log entry in our CMS. Just a dropped connection that we only discovered by checking raw server access logs. Fixing it required a WAF configuration change, not a robots.txt edit.
That’s one of five common reasons businesses don’t appear in ChatGPT’s answers. Here’s the full diagnostic.
1. AI Crawlers Are Blocked (and You Might Not Know It)
This is the most common cause and the hardest to detect without looking in the right place.
There are two layers where AI crawlers can be blocked: robots.txt and your WAF/CDN. Most people check robots.txt and stop there. That’s a mistake.
Robots.txt blocks are the obvious ones. If your robots.txt file includes “User-agent: GPTBot” followed by “Disallow: /”, ChatGPT’s training crawler can’t index your site. Same pattern applies to OAI-SearchBot (ChatGPT’s browsing crawler), ClaudeBot, PerplexityBot, and Google-Extended.
WAF/CDN blocks are the invisible ones. Your Cloudflare, Sucuri, Akamai, or AWS WAF settings may classify AI crawlers as bots and block them before the request reaches your server. The telltale sign is a 000 status code or connection timeout in your access logs for known AI user-agents.
CROSS Digital’s WAF Discovery: A Case Study
Here’s exactly what happened on our own site.
When we started building our GEO strategy in early 2026, we ran a crawler access audit. Our robots.txt was clean. No AI crawlers were blocked there. But when we pulled our raw server access logs and filtered by user-agent, the picture was different.
GPTBot and Google-Extended requests were returning a 000 status code. That’s not a standard HTTP status. It means the connection was dropped at the network edge before our server processed the request. Our WAF was treating these crawlers as suspicious bots and silently rejecting them.
Meanwhile, ClaudeBot and PerplexityBot were getting 200 status codes. They were reaching our server fine. Same website, same content, but different AI crawlers getting different treatment based on WAF rules we hadn’t specifically configured.
The fix wasn’t in robots.txt. We had to adjust our WAF’s bot management rules to whitelist GPTBot and Google-Extended. Once we did, those crawlers started returning 200 status codes, and our content became accessible for ChatGPT’s browsing mode and Google’s AI training pipeline.
We documented the full technical details in our Generative Engine Optimization Guide. The key lesson: robots.txt is only half the access story. Your infrastructure layer can override it entirely.
2. Your Site Relies on JavaScript-Only Rendering
AI crawlers are generally less capable at rendering JavaScript than Googlebot. If your site’s primary content loads via client-side JavaScript (React, Angular, or Vue without server-side rendering), AI crawlers may see a blank or nearly empty page.
Check this by viewing your page’s source code (not the rendered DOM in DevTools, but the actual HTML returned by the server). If your main content isn’t in the raw HTML, AI crawlers likely can’t see it either.
The fix is server-side rendering (SSR) or pre-rendering for critical content pages. At minimum, ensure your homepage, service pages, and key blog posts deliver their text content in the initial HTML response.
3. Weak Entity Consistency Across the Web
AI models cross-reference multiple sources when deciding whether to mention a business. If your company name, description, or details vary across the web, the model has lower confidence in your entity.
Common inconsistencies we see: different business names on Google Business Profile vs. your website vs. LinkedIn (“CROSS Digital” vs. “Cross Digital Marketing” vs. “CROSS DMA”). Different phone numbers. Different addresses. Different descriptions of what the company does.
Each inconsistency erodes confidence. AI models prefer to cite businesses they can verify across multiple matching sources.
4. No Corroboration from Independent Sources
If the only detailed information about your business exists on your own website, AI models have just one source to draw from. That’s a weak signal.
Corroboration means independent sources confirming your business exists and does what you say it does. These include press mentions, guest articles on industry publications, directory listings, partner case studies, review sites, and industry association pages.
Think of it from the model’s perspective. If five independent websites describe your company as “a digital marketing agency in Burlington, MA specializing in SEO and paid media,” the model has five data points confirming that entity. If only your homepage says it, the model has one. More corroboration means higher confidence, which means more frequent citations.
5. Stale or Outdated Information in AI Training Data
ChatGPT’s base knowledge comes from training data with a cutoff date. If your business launched after that cutoff, pivoted its services, changed its name, or moved locations, the model’s training data may not reflect your current reality.
You can’t directly update ChatGPT’s training data. But you can ensure your current information is available through browsing mode by keeping your site accessible to OAI-SearchBot and maintaining accurate, up-to-date content. When ChatGPT browses the web to answer a query, it can find and cite your current information even if its training data is outdated.
Structured data helps here too. Schema markup gives AI crawlers a machine-readable summary of your business that’s easier to parse than unstructured page copy. Our guide on schema markup for AI search [Internal Link: /blog/schema-markup-ai-search] covers implementation specifics.
Self-Diagnostic Checklist
Run through these checks in order. Each one builds on the previous.
| Check | What to Look For |
| robots.txt review | Visit yourdomain.com/robots.txt. Confirm GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended aren’t blocked. |
| WAF/CDN audit | Check your CDN or WAF bot management settings. Look for blanket bot-blocking rules that might catch AI crawlers. |
| Access log analysis | Filter server logs by AI user-agents. Look for 000 status codes, 403 errors, or connection timeouts. |
| JavaScript rendering test | View page source (not DevTools). Confirm key content appears in the raw HTML. |
| Entity consistency audit | Search your business name across Google, Bing, LinkedIn, Crunchbase, Yelp, and industry directories. Note any variations in name, address, phone, or description. |
| Schema markup check | Verify your site has Organization, LocalBusiness, or relevant schema markup. Use Google’s Rich Results Test. |
| Corroboration inventory | List every independent website that mentions your business. Aim for at least 8 to 10 corroborating sources. |
| ChatGPT prompt test | Ask ChatGPT 10 category-relevant prompts. Record which businesses appear and whether yours is mentioned. |
| Competitor comparison | Run the same prompts and note which competitors consistently appear. Investigate what they’re doing differently. |
| Monthly monitoring | Set a calendar reminder to repeat the prompt test monthly. Track changes over time. |
Robots.txt vs WAF: Understanding the Difference
This distinction trips up most site owners, so it’s worth spelling out clearly.
Robots.txt is a text file on your web server that tells crawlers which URLs they’re allowed or not allowed to visit. It’s a voluntary protocol. Well-behaved crawlers read and respect it, but technically any bot can ignore it. Robots.txt is visible to anyone at yourdomain.com/robots.txt.
A WAF (Web Application Firewall) sits between your server and the internet. It inspects incoming requests and blocks ones that match its threat rules. Unlike robots.txt, a WAF block isn’t voluntary. The request gets dropped or rejected before it ever reaches your server. The crawler gets no content and often no useful error message.
The practical problem: you can have a perfectly configured robots.txt that allows all AI crawlers, while your WAF silently blocks them. Checking only robots.txt gives you a false sense of security. You have to check both layers.
Frequently Asked Questions
Related guides
The Complete Guide to Generative Engine Optimization (GEO)
How to Get Your Business Mentioned in ChatGPT
Want CROSS Digital to Diagnose Your AI Visibility?
We’ve run this exact diagnostic on our own site and fixed every issue on this list. Our AI visibility audits give you a clear picture of where you stand across ChatGPT, Perplexity, Claude, and Google AI Overviews, with a prioritized action plan to fix what’s broken.
CROSS Digital Marketing Services
Technology & Integration
Web Hosting Service
Artificial Intelligence
We’re the team you’ve been looking for
We’d love to hear about your project and learn how CROSS Digital can partner with you.