AI Tools

How to Block AI Crawlers (GPTBot, ClaudeBot, PerplexityBot) in 2026

Published 2026-08-13 · Researched and price-checked on official sources

How to block AI crawlers in 2026 starts with an uncomfortable fact: robots.txt is a voluntary request, not enforcement. AI-related bots now make up 29% of all bot traffic, and attackers are spoofing GPTBot and ClaudeBot to scan sites for weaknesses. Enforce at the edge with Cloudflare's free policies, and add the AI Labyrinth honeypot.

Key facts

What matters

AI-related bots now make up 29% of all bot traffic on the web, up 11% versus the prior 90 days, per knownagents.com's Agentic Web Index (checked August 13, 2026).
robots.txt blocks nothing: Cloudflare's docs describe compliance as voluntary, and Cloudflare caught Perplexity using stealth crawlers that ignored no-crawl directives.
Free Cloudflare options cover most sites: managed robots.txt on all plans, AI bot blocking policies for all customers, and the AI Labyrinth honeypot that traps unauthorized crawlers.
User-agent blocklists are spoofable: knownagents.com is tracking a live campaign that impersonates AI bots to scan websites for vulnerabilities, so verify identity rather than trusting a UA string.

What counts as an AI crawler in 2026?

An AI crawler is a bot that visits websites to feed AI systems. Cloudflare's bot documentation sorts them into three behaviors: Search (crawlers that collect content to answer questions about it later, like PerplexityBot), Agent (automated activity acting in real time for a person, such as chat fetch bots), and Training (crawlers taking content to train or fine-tune a model, like GPTBot).

They are a large and growing share of web traffic. The Agentic Web Index from knownagents.com, the operator of Dark Visitors, shows bots make up about 35% of website visits, and 29% of that bot traffic is AI-related, up 11% versus the previous 90 days. The most active AI crawlers it tracks include GPTBot (OpenAI), PerplexityBot, ClaudeBot (Anthropic), Bytespider (ByteDance), and meta-externalagent (Meta).

One more category matters for blocking: spoofed crawlers. A visit counts as spoofed when it claims a recognized agent identity but fails that agent's authentication method, such as verified IP ranges or Web Bot Auth, per the index's methodology.

Why blocking AI crawlers matters right now

The stakes moved from bandwidth to business. The Walrus reported on August 10, 2026 that AI-generated answers are pulling clicks away from the open web: Wikipedia's traffic is dwindling, and news organizations have started blocking the Wayback Machine's crawlers over fears that archived pages give AI companies an indirect source of copyrighted material.

knownagents.com is also tracking an active threat it calls an AI Bot Spoofing Campaign: a widespread campaign impersonating AI bots to scan websites for vulnerabilities, apparently targeting credential and configuration paths used by AI coding tools. If you block crawlers by user-agent string alone, that campaign walks past your rules, because the attackers simply claim a known AI identity.

This is why the how-to changed in 2026: preference files plus identity-verified enforcement, not a static blocklist.

Method 1: robots.txt (a request, not a wall)

robots.txt is a text file at the root of your domain that tells crawlers what they may access. It is the cheapest first step, and it stops well-behaved crawlers. Cloudflare's managed robots.txt setting, available on all plans, generates and maintains this file for you, adding Disallow: / rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent. It also appends content signals, search=yes, ai-train=no, use=reference, which reserve your content against AI training while keeping it indexable for search.

Here is the limit: compliance is voluntary. Cloudflare's own docs say robots.txt does not prevent crawlers from accessing your content at a technical level, and some operators crawl regardless. Cloudflare documented Perplexity using stealth, undeclared crawlers that evaded no-crawl directives, then de-listed Perplexity as a verified bot and added heuristics to block that stealth crawling. If you want to audit which user agents hit your site, Dark Visitors (now part of knownagents.com) publishes a free AI bot robots.txt checker and an agent directory.

Method 2: how to block AI crawlers with Cloudflare's free policies

Cloudflare's Block AI bots setting blocks verified bots classified as crawling for AI training, plus unverified bots that behave the same way, and it is available to all Cloudflare customers. In mid-2026 Cloudflare restructured this into per-behavior policies, letting you treat Search, Agent, and Training traffic differently, with three mitigations each: block everywhere, block only on pages with ads, or allow.

New defaults arrive September 15, 2026: bots classified as Training or Agent will be blocked on pages that display ads, Search stays allowed, and mixed-purpose crawlers that combine Search and Training get blocked by every configuration that blocks AI training, including the legacy option. If you want different treatment, opt out in Security Settings before that date.

Method 3: AI Crawl Control and the AI Labyrinth

AI Crawl Control takes per-crawler action. The dashboard lists each AI crawler requesting access, its category, its request volume, and how many times it has violated your robots.txt file, so you can allow, block, or configure each one individually.

AI Labyrinth, described in Cloudflare's docs and announced on its blog, is a honeypot: it adds invisible links with nofollow tags that trap AI crawlers ignoring no-crawl rules in a maze of never-ending pages. The links do not affect SEO or page appearance, and AI bots that respect no-crawl instructions safely ignore the honeypot. Cloudflare's blog notes the links are presented only to suspected AI scrapers, so regular visitors and verified crawlers browse normally. Enable it in Security Settings under Bot traffic.

The catch: spoofed bots make user-agent lists useless

Every blocklist approach shares one hole: user agents are self-reported. The spoofing campaign knownagents.com is tracking proves the point. Cloudflare verifies bots with IP address ranges, reverse DNS records, and message signatures, and a visit should count as spoofed when it fails those checks. If you maintain your own blocklist, pair it with verification at the edge, such as Cloudflare or a WAF that checks verified-bot signals, rather than trusting the UA string in your logs.

A practical rule: treat robots.txt as your preference, treat identity-verified blocking as your enforcement, and treat honeypots as your defense against crawlers that ignore both.

Should you block AI crawlers? The trade-off

Blocking has a cost. AI chat referrals, visits that arrive from an AI answer citing your page, currently represent 0.1% of human website visits in the Agentic Web Index. For a publisher selling content, hiding from Training crawlers protects your material from being absorbed into models. For a business that wants to be cited in AI answers, blocking Search-classified crawlers like PerplexityBot cuts off a small but growing referral channel.

The balanced default, which matches Cloudflare's September 2026 policy, is to block Training and Agent traffic while allowing Search. Google Search Console may flag the newer directives as Syntax not understood, but Cloudflare says it has observed no impact on crawling rates or SEO.

Where to start: a practical order of operations

Audit who is already crawling you: check the AI Crawl Control crawlers tab in Cloudflare, or knownagents.com's index, and note the robots.txt violation counts.
Turn on Cloudflare's managed robots.txt (all plans) to publish Disallow rules for known AI crawlers plus content signals like ai-train=no.
Set per-behavior policies in Security Settings: block Training and Agent, keep Search allowed, or opt out of the September 15, 2026 defaults if your site depends on AI search visibility.
Enable AI Labyrinth so crawlers that ignore no-crawl rules burn time in the honeypot instead of your real pages.
Treat any user-agent blocklist you maintain as a supplement, not the defense: spoofed identities will bypass it.

At a glance

Approach	What it does	Enforcement	Availability
Managed robots.txt	Serves Disallow rules for known AI crawlers plus content signals (search=yes, ai-train=no)	Voluntary: compliant crawlers honor it	All Cloudflare plans
Block AI bots policy	Blocks verified Training bots plus similar unverified bots	Enforced at the edge	All Cloudflare customers; new defaults September 15, 2026
AI Crawl Control	Per-crawler allow or block, with a robots.txt violation counter	Enforced at the edge	Cloudflare dashboard feature
AI Labyrinth	Invisible honeypot links that trap crawlers ignoring no-crawl rules	Trap; invisible to humans, no SEO impact	Cloudflare, toggle in Security Settings

Four ways to block AI crawlers, verified against Cloudflare's documentation on August 13, 2026

FAQ

Does robots.txt actually stop AI crawlers?

No. robots.txt expresses permission, and compliance is voluntary. Cloudflare documented Perplexity evading no-crawl directives with stealth crawlers, and it maintains AI Crawl Control and AI bot policies precisely because preferences are not enforcement. Use robots.txt as a signal and pair it with identity-verified blocking.

How do I block GPTBot, ClaudeBot, and PerplexityBot?

The quickest route is Cloudflare's managed robots.txt setting, which adds Disallow rules for GPTBot, ClaudeBot, Amazonbot, Bytespider, CCBot, Google-Extended, and others on all plans. To enforce rather than request, turn on the Block AI bots policy or AI Crawl Control. Note that attackers spoof these identities, so verify bot identity (Cloudflare checks IP ranges, reverse DNS, and message signatures) instead of trusting user-agent strings.

Will blocking AI crawlers hurt my SEO or AI search visibility?

Cloudflare reports no impact on Google crawling rates or SEO from its managed robots.txt, even when Search Console shows a Syntax not understood warning. The real trade-off is AI search visibility: AI chat referrals are currently about 0.1% of human visits, and Cloudflare's September 2026 defaults block Training and Agent bots while keeping Search crawlers allowed, which is a sensible middle ground.