The AI Crawler Extractability Index
what B2B sites actually serve a crawler that does not run JavaScript

This page is a measurement, so it carries a method, a date, and a commitment to re-run.

What does it mean for a site to be invisible to an AI crawler?

A site is invisible to an AI crawler when the HTML in the first response contains none of the page's actual words, and the crawler executes no JavaScript that would put them there. Not slow. Not badly ranked. Empty.

Why does everything you already measure say the page is fine?

Every check most teams run renders JavaScript, which means every check most teams run answers a question nobody asked. The failure appears only in a raw fetch with a bot user agent, and that is the one test nobody has in their weekly routine.

How do you measure emptiness without guessing at a baseline?

You measure your own empty shell first, then read every route against it. Request a URL on your domain that cannot exist. If a catch all route returns 200, that response is your shell, measured and not assumed, and every byte count on the site becomes readable.

I did not arrive here from web development. I arrived from running campaigns, where the first thing anybody teaches you is that a percentage with no denominator beside it cannot be acted on. The group publishes its campaign figures on the properties that own them, always with the denominator in the same sentence, and this page does not restate them, because a figure restated on four domains stops being one citable asset and becomes four weaker ones.

Byte counts behave exactly the same way. 5,519 bytes tells you nothing at all. 5,519 bytes read against a shell measured on the same domain on the same day at 5,179 bytes tells you the page holds one heading and one sentence. The shell measurement is the denominator, and every figure on this page travels with it.

We run crawls the same two ways we run campaigns, and use the same two names. WideNET sweeps every route type, because coverage gaps hide in routes nobody thought to check. Spearhead goes straight at the handful of URLs a buyer actually gets sent to. WideNET gives you the shape of the problem, Spearhead tells you whether it is expensive.

What did we measure on our own site?

Read this table as a dated worked example, not as a current state. It was measured on 6 August 2026. Extending the parent's prerender coverage to the failing routes was already scheduled when this was written and is a precondition of publishing anything else in the group, so by the time you read this the right hand column has probably changed. That is the point. The measurement is here to show you what the method finds and what the numbers look like when a real site fails.

The re-measurement commitment, in writing. The same eight routes get fetched again with the same command on the day the prerender fix ships, and the result gets published here as a second column beside this one, whichever way it goes. Both columns stay. Deleting the failing column and presenting the fixed one would turn a measurement into a case study, and the delta between the two is the more useful dataset anyway, because nobody else can publish a before and after on the same domain with the same instrument. The fix date is not set as this publishes, so this page carries one column until it is.

Which pages fail most often, the blog or the homepage?

On our own site the pattern inverted the one everybody expects: the editorial routes passed and the commercial routes failed. Prerendering had been written for the blog, reviews and directory route patterns, and had never been extended to the homepage, the tool pages or the service pages.

Publishing that about a property in your own group is uncomfortable and it is the reason the finding is worth anything. A vendor who will not name a failure on a site they control is not going to name one on yours.

If the wider aggregate reproduces that shape, it is the most useful thing this index will produce, because it means the standard advice to fix your blog first is backwards.

How many B2B sites fail this test?

We are not publishing a percentage yet, and the reason is the point of this page. We do not have a sample to compute one from: the checker runs entirely in the reader's browser and sends us nothing, so no run reaches us at all. Even if it did, the sample would be self selected, because people who put a URL into an AI visibility tool already suspect they have a problem. A percentage from that sample would overstate the failure rate, and it would travel further than its caveat.

Here is the rubric the aggregate publishes against, fixed in advance so it cannot be tuned later into a better headline. Every URL is banded against its own domain's shell, never a fixed byte threshold.

The two boundaries below are editorial design decisions, not measured coefficients. Nobody has published a study establishing the byte count at which a page becomes usefully quotable, and we have not run one either. We chose 500 and 3,000 because they separate the three states cleanly against real shells we have measured, and we are publishing them so you can disagree with them by hand. They are frozen before the first aggregate ships and they do not move afterwards, because a boundary adjusted after seeing the distribution is a headline, not a threshold.

Category cuts ship only where the cell size clears a minimum, and that minimum gets set and written down before the first aggregate publishes rather than after, so it can never be lowered to make a cut publishable.

How do you run this on your own site in five minutes?

Four commands, in order, and the order matters because step one is what makes steps two through four readable.

Step 1. Measure your empty shell

Request a path that cannot exist. If your app returns 200 for it, that byte count is your baseline.

curl -s -A "GPTBot" "https://yourdomain.com/this-route-does-not-exist-9471" | wc -c

Step 2. Sweep every route type against it

for u in "/" "/pricing" "/blog" "/blog/some-post" "/product" "/about"; do
  printf "%-24s " "$u"; curl -s -A "GPTBot" "https://yourdomain.com$u" | wc -c
done

Anything within a few hundred bytes of your baseline is empty. Anything several times larger is real. Partial coverage shows up in this list immediately and almost never shows up any other way.

Step 3. Read the body, not just the size

Look for your actual words. Not the title tag, not the meta description, not the JSON-LD. Your H1, and a sentence from the middle of the page.

Step 4. Repeat with the other documented agents

OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. A 403 or a challenge page means the block sits at your CDN rather than in your build, which is a different fix.

Does fixing this change anything you can actually see?

We do not know yet, and anyone telling you otherwise is describing a model, not a measurement. No first party citation reporting exists for ChatGPT, Claude or Perplexity, and referral traffic undercounts badly because native app traffic sends no referrer header.

On the campaign side of the group every kill and scale threshold is written down before the first email leaves, because a threshold agreed after seeing the numbers is a rationalisation wearing a threshold's clothes. The thresholds themselves are published on the group property that owns rate definitions, with the denominator attached, and this page does not restate them.

I want that discipline here and do not have it yet, so here is what we pre-committed to instead of what we hope happens. Before the parent's prerender coverage extends we record three baselines: the Bing Webmaster Tools AI Performance report, the only free first party per URL citation data any platform offers and the one place the grounding queries appear; the Search Console generative AI report for Google's surfaces; and a frozen panel of identical questions asked monthly of each assistant in fresh sessions. Then we extend the prerender, keep both the before and after rows instead of replacing them, and publish whatever the panel says, including the runs where nothing moved.

What does this index not measure?

This index measures one thing: what a non rendering crawler receives in the first response, on one fetch, at one moment. Six limits, because a benchmark that does not mark its own edges is a marketing asset pretending to be data.

• No citation data. Passing means an assistant could read you, not that one quoted you.
• Not Googlebot’s view. Googlebot renders, so a site can fail this test completely and keep its classic rankings.
• No corroboration signal. Retrieval resolves entities by agreement across sources, and your own site is the weakest surface for exactly the commercial queries that convert. This index cannot see that at all.
• No crawl frequency. Nobody publishes how often these agents return.
• Nothing behind a login. Public routes only.
• No JavaScript injected schema. Structured data written by a script is not in the payload we counted, which is rather the point.

A seventh limit sits outside the method. This measures whether you can be retrieved, not whether enough of the right buyers know your category exists yet. That second problem is an outbound question rather than a retrieval one, and it has a different budget and a different owner.