Do AI crawlers respect robots.txt?
What the logs show, bot by bot

Why is the honest answer bot specific?

Because robots.txt is a voluntary standard. A compliant crawler can read it and follow it. A non compliant crawler can ignore it. And a spoofed user agent can pretend to be something reputable while doing whatever it wants. If you are trying to decide whether AI systems can access your content, the useful question is not whether robots.txt works in theory. It is which fetchers hit your site, how they identify themselves, whether they request robots.txt first, and what changes after you block them.

That is why I prefer logs over vendor pages. Documentation tells you intended behavior. Logs tell you actual behavior on your infrastructure. For AI visibility work, that distinction matters because extraction depends on what a fetcher can access, not what a help page says should happen.

What patterns in logs suggest a crawler respects robots.txt?

There are a few practical signals. First, the crawler requests /robots.txt before or alongside deeper page fetches. Second, after a new disallow rule appears, the crawler stops requesting the blocked paths or materially narrows what it asks for. Third, the crawler keeps accessing allowed assets while avoiding blocked sections. That is much stronger evidence than a single robots.txt fetch.

The opposite pattern is also revealing. If a bot repeatedly requests blocked paths after fetching robots.txt, either it is ignoring the rules, the rule is malformed, or the traffic is not really from the bot it claims to be. Attribution is the hard part. Logs show requests. They do not automatically prove corporate ownership of the requester.

• Useful signal: a robots.txt fetch appears immediately before content fetches
• Stronger signal: blocked directories stop receiving requests after a disallow
• Weak signal: the user agent contains a familiar bot name, with no corroboration
• Bad assumption: one request to robots.txt means the crawler will obey it forever

What do the main bot categories usually look like in practice?

I group them into three buckets. The first is named AI crawlers with stable documentation and recognizable user agents. The second is search or platform infrastructure that may contribute to AI answers indirectly but does not market itself as an AI crawler. The third is unverified traffic that borrows AI flavored user agents. The first bucket is where robots.txt analysis is most actionable. The third bucket is where teams waste time unless they validate carefully.

This is where teams want a clean ranking of good bots and bad bots. You usually cannot do that honestly from a single site log set. Different sites have different server setups, cache layers, bot management rules, and visibility levels. A bot that looks well behaved on one property may barely appear on another because it has no reason to fetch much there.

How should you read server logs without fooling yourself?

Start with method discipline. Pull raw requests for robots.txt, then isolate user agents associated with AI crawlers or AI adjacent fetchers. Look at sequence, not just totals. Did the crawler ask for robots.txt before content? Did behavior change after your file changed? Did blocked paths stop? Then sample request headers, IP consistency, and repeat patterns over time. One day of traffic is anecdote. Repeated sequences are evidence.

Also remember a key verified point from recent log based work: AI crawlers fetch JavaScript files and do not execute them. That matters here because teams sometimes think a robots block on rendered routes protects content that only appears after client side execution. It does not work that way. If your critical text is hidden behind JavaScript, many AI crawlers will not render it anyway. Your robots policy and your rendering model are separate controls.

Where does robots.txt fail as an AI visibility control?

It fails when people expect it to do jobs it cannot do. It does not authenticate a crawler. It does not prevent scraping by actors who ignore standards. It does not cleanly separate training, indexing, retrieval, summarization, and citation use cases unless the crawler owner has implemented those distinctions. And it does not tell you whether your content is being quoted from another copy on the web.

It also fails when your internal goal is fuzzy. If you want more AI citations, blocking broad sections can reduce extractability. If you want less AI access, robots.txt may help with compliant crawlers but not with everyone else. Many teams try to hold both positions at once, then blame the file format. The real issue is policy ambiguity.

• Robots.txt can guide compliant crawlers
• Robots.txt cannot force malicious or spoofed traffic to stop
• Robots.txt cannot fix poor extractability on pages that are hard to quote
• Robots.txt cannot substitute for log analysis and page level testing

What should you actually do if you want evidence, not folklore?

Run a simple controlled test. Pick a low risk section of your site. Document current requests from a named crawler. Change robots.txt for that section. Watch whether the request pattern changes over a clean observation window. Then reverse the rule if needed. You are not trying to prove platform wide truth. You are trying to answer an operational question about your own stack and your own exposure.

Pair that with page design work. If a page is meant to earn citations, make the core claim visible in server rendered HTML, use clear entities, and keep supporting evidence close to the claim. If a page should not be an easy extraction source, do not assume a robots line is enough. Think in layers: access control where warranted, selective exposure, and realistic expectations about what public pages can still leak through mirrors, references, and summaries elsewhere.

If your problem is outbound execution or prospecting mechanics, that belongs on the parent brand, not here. We run managed outbound under Outbound Pros, but this site is for demand capture and AI search. The overlap is real, the operating motions are not the same.

Who should not follow this advice as written?

If you run a tiny brochure site and you have not yet established whether AI systems can quote your important pages at all, do not start with robots policy experiments. Start with extractability and crawlable HTML. Blocking before you can measure is just moving fog around.

If you are in a regulated environment with strict data controls, do not rely on robots.txt as your main safeguard. It is a courtesy protocol, not a hard perimeter. Use stronger controls. And if your traffic is heavily filtered by a CDN or bot management layer, be careful with conclusions from partial logs. You may be seeing what your edge lets through, not the full request landscape.

This is the trade off most posts skip. The more open and extractable your pages are, the easier they are to cite. The same openness can make you less comfortable if your instinct is to block AI systems broadly. There is no universal right answer. There is only a deliberate policy with evidence behind it.