Legible vs. Cloudflare AI Blocking: Optimize Content for AI or Block Bots?

Opposite approaches to AI crawlers

There are two camps when it comes to AI crawlers: block them, or make your content easier for them to read. Cloudflare and Legible sit on opposite sides of that line. See our detailed comparison for a full breakdown.

What Cloudflare's AI blocking does

Cloudflare sits in front of roughly 20% of all websites. They added a feature that lets site owners block known AI crawlers by user-agent. One click, and GPTBot, ClaudeBot, and others can't index your content at all.

For some use cases, this makes sense. If you're running a paywall publication and don't want AI systems to give away your premium content for free, blocking is a reasonable defensive posture.

The problem with blocking

The internet grew by being open. Content that could be indexed, linked, and cited accumulated more influence over time. AI-mediated search follows the same pattern. Content that AI systems can access shows up in AI answers. Blocked content doesn't.

If you block AI crawlers today, you're opting out of a distribution channel that's growing fast. When someone asks ChatGPT "what's the best tool for X?", the answer will cite accessible content, not yours.

What Legible does instead

Legible takes the opposite approach. Instead of blocking AI access, we make your content easy for AI to read and use correctly. We strip the HTML noise, serve clean Markdown, maintain a live llms.txt, and give you granular permission controls over what AI can do with your content.

The result: AI systems get a clean, accurate, citable version of your content. They're more likely to include it in answers, attribute it correctly, and send users to your site.

Which approach is right for you?

If you're protecting premium, paywalled content, Cloudflare's blocking makes sense. But if your goal is visibility and reach, blocking AI crawlers works against you. Legible is built for that second group — get started today.