How to Block AI Crawlers and Protect Your Website Content in 2026

Site update (August 2026): DevKitDock now uses a middle-ground policy. Search and AI citation bots (Google, Bing, GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are allowed so AI engines can find and cite our content. Only training-only crawlers such as CCBot, Bytespider, cohere-ai and Amazonbot are blocked. The blocking techniques below still work if you decide your content should stay out of AI indexes entirely.

August 6, 2026 · 6 min read · by DevKitDock Editorial

AI companies are scraping the web at an unprecedented scale. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and ByteDance's Bytespider are crawling millions of websites every day — often ignoring standard etiquette — to train their next-generation language models.

If you publish original content — blog posts, tool descriptions, code examples, design assets — your work might already be feeding someone else's AI without your consent. The good news: blocking them is straightforward, and you can do it without hurting your Google rankings.

Why This Matters Now

In 2025-2026, the AI scraping landscape changed dramatically. Several high-profile incidents revealed that major AI companies were:

Ignoring robots.txt directives entirely
Spoofing user-agent strings to bypass blocks
Consuming bandwidth equivalent to DDoS-level traffic on small sites

While not all AI crawlers misbehave, the default posture for independent publishers has shifted from "allow by default" to "block by default, allow selectively."

Step 1: Identify AI Crawler User Agents

Here are the major AI crawlers active as of mid-2026:

User Agent	Company	Purpose
`GPTBot`	OpenAI	Training GPT models
`ClaudeBot`	Anthropic	Training Claude models
`PerplexityBot`	Perplexity AI	Search index & training
`Bytespider`	ByteDance	Training Doubao & other models
`Google-Extended`	Google	Gemini training (separate from search)
`CCBot`	Common Crawl	Public web corpus (used by many AI labs)
`cohere-ai`	Cohere	Training Command models
`Amazonbot`	Amazon	Alexa & Bedrock training

Step 2: Write Your robots.txt

The key principle: allow search engines, block AI scrapers, deny everything else. Here is a production-ready configuration:

# robots.txt — Allow search, block AI scrapers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Amazonbot
Disallow: /

# Deny everything else by default
User-agent: *
Disallow: /

Sitemap: https://devkitdock.com/sitemap.xml

Important: Google Search vs. Google AI

Notice the distinction between Googlebot and Google-Extended. Googlebot crawls for search indexing — you want this. Google-Extended crawls for Gemini/Bard training — you probably want to block this. They are separate crawlers with separate robots.txt rules.

Blocking Google-Extended does NOT affect your Google Search rankings. Google has confirmed this in their documentation.

Step 3: Deploy and Verify

Upload your robots.txt to your website root. On Cloudflare Pages, place it in your build output directory. Verify it works:

Visit https://yourdomain.com/robots.txt — you should see the file contents
Use Google Search Console's robots.txt tester to validate syntax
Check your server logs after 48 hours — AI crawler hits should drop significantly

What robots.txt Cannot Do

Be realistic about what robots.txt provides. It is a voluntary protocol — well-behaved crawlers respect it, but malicious ones ignore it. For stronger protection, consider:

Cloudflare WAF rules — Block AI bot user agents at the edge
Rate limiting — Cap requests per IP to prevent scraping floods
JavaScript challenges — Force non-browser clients through a JS check

Summary

Blocking AI crawlers takes five minutes and a properly configured robots.txt. The strategy is simple: green-light Google and Bing for search traffic, red-light everyone else by default, and specifically name the AI crawlers you want to keep out. It is not bulletproof, but it is the first and most important line of defense for independent publishers in 2026.

Related tools: Diff Checker · Text Counter · Hash Generator

💬 Comments (0)

Detecting IP...