The Bouncer at the Door: How to Check (and Fix) Whether AI Can Actually Get Into Your Site
You could write the most quotable, perfectly structured, schema-marked-up page on the internet. If there's a bouncer standing at your front door turning away every AI crawler before it gets inside, none of it matters. The model never reads it. It never gets cited. It never gets recommended.
We've talked about why GEO matters and what makes a business get picked ("The Librarian vs. The Concierge," "The New Word of Mouth"). This one's about the step nobody checks first: can AI actually get in the door at all?
You're not being paranoid — nearly 1 in 5 of the top 100,000 sites on the web are blocking at least one major AI crawler, often without realizing it. GPTBot alone is blocked by over 15% of sites (USTA, July 2026).
The Two Files Everyone Confuses
There are two small text files on almost every website that people assume do the same job. They don't.
robots.txt is the one with actual authority. It's a real instruction that crawlers are expected to honor — it can say "come in" or "stay out," page by page, bot by bot. If it's misconfigured, it can genuinely block AI crawlers like GPTBot, ClaudeBot, or PerplexityBot from reading your site at all.
llms.txt is often assumed to work the same way. It doesn't. It has zero enforcement power — it's a plain-text index that some AI coding tools use to find documentation, and that's about it. It can't block anything, can't allow anything, and can't cause an error. A broken or missing llms.txt does nothing. If your site isn't showing up in AI answers, this file is not why.
The takeaway: if you're going to check one thing, check robots.txt. Don't waste time on llms.txt thinking it's a visibility lever — it isn't.
Three Ways to Actually Check — Instead of Guessing
1. Use OpenAI's own checker. Go to openai.com/gptbot, enter your URL, and it will tell you directly whether GPTBot can access that page. Ten seconds, no setup required.
2. Run a direct test yourself.
curl -I -A "GPTBot" https://yourdomain.com/your-page
This sends a request pretending to be GPTBot and shows exactly what comes back. On Windows, PowerShell's built-in curl is actually a different command in disguise and will error on this — use curl.exe instead, or the PowerShell-native version:
Invoke-WebRequest -Uri "https://yourdomain.com/your-page" -Method Head -UserAgent "GPTBot"
3. If you're on Squarespace, check one specific setting. Under Settings → Crawlers, there's a checkbox called "Block known artificial intelligence crawlers." It's unchecked by default — Squarespace doesn't block AI crawlers out of the box — but confirm nobody flipped it on thinking it was a safety switch. Other platforms have an equivalent bot-management setting somewhere in security/firewall settings.
Reading the Result
A 200 means the page loaded successfully — the crawler got in. That's what you want.
A 403 means access was actively refused — something is explicitly blocking that crawler, usually a robots.txt rule or a bot-blocking setting.
A 429 means "too many requests," and it's worth a caveat: it doesn't necessarily mean a permanent block. It can just mean the request tripped a rate limiter, which happens if the same page gets hit repeatedly in a short window. Don't treat a single 429 as proof of a real problem — check that the page loads normally in a browser, and re-run the test once, cleanly, before assuming the worst.
The Two-Minute Audit
Visit yourdomain.com/robots.txt directly and read what it actually says.
Run the GPTBot checker on your two or three most important pages.
If you're on Squarespace, confirm the "block known AI crawlers" box is unchecked.
If anything comes back blocked, fix the rule, then re-test — don't assume it's fixed until you've confirmed it.
It's a smaller check than writing a single paragraph of content. Skipping it means every other GEO effort is being built on top of a door that might not open.
written by The Team at Danu Agency
Not sure if your site passes this audit — or want someone to run it for you and fix what's broken? Book a free consultation and we'll check.
