The 2026 Thales and Imperva Bad Bot Report put automated traffic at 53% of the web in 2025, with human traffic at 47%. That makes user-agent strings increasingly unreliable as the only control. A global deny is a pricing decision dressed as a security policy. A Rutgers and Wharton working paper (revised April 2026) found large news publishers that blocked GenAI bots in robots.txt lost about 7% of weekly visits within six weeks, including in household browsing panels, not just bot metrics.
The usual map names a crawler list, a robots.txt deny, and edge challenges. Three controls actually change the bill: return 402 or 403 to training bots, challenge spoofed agents with signed identity, and measure who is citing you before you blanket-block. An allowlist names GPTBot, the search fetchers, and the partners you have a deal with. A blocklist names the training bots and hopes the rest behave. Those are two controls. The crawler you want is the one you named.
We compared six tools on that split: whether you are buying analytics or an edge deny, whether the deny sits on a CDN you already run, and whether any price is public. If you need per-bot analytics and a public self-serve price before you write a rule, start with Known Agents. If the site already sits on Fastly and you want inspection in front of cache, Fastly’s crawler controls are the most natural edge-layer option. If you want per-crawler allow / block / charge on a zone you already run on Cloudflare, AI Crawl Control is the dashboard you already have. If the same article URLs, the subscription API, and the app need one bot policy, Radware is the full-stack product. If the site already sits on Akamai, Bot Manager is the edge score-and-respond path. If you already run Imperva, Advanced Bot Protection is the WAAP layer for scrapers and logins.
What usually goes wrong when buying AI crawler controls
Most quotes in this category fail for the same few reasons. The fix is operational, not a new acronym.
| Problem | Solution |
|---|---|
| You bought a report when you needed a deny | Write whether the product measures or enforces |
| The deny does not sit on an edge you already run | Match the control to the CDN or edge you have |
| AI-bot slides hide the price | If the vendor publishes a price for the crawler-control product, use that rate. Otherwise treat pricing as custom |
| An allowlist exception is missing from the contract | Ask how a good crawler gets through without opening the rest |
How we evaluated AI crawler controls
We compared whether each product measures or enforces crawler behavior, where enforcement runs, whether pricing is public, and whether it supports actions beyond robots.txt. robots.txt-only controls and a second CDN you do not run sit in What we left out because they do not clear those filters.
TL;DR: The 6 Compared
| Service | Best for | What to check |
|---|---|---|
| Known Agents | Per-bot analytics and a public self-serve price before you write a deny |
ControlAnalytics. Names the bot, the page, and the LLM referral. Does not sit on the CDN edge
PriceFree $0 (1 project, 100K events). Startup $29 (3 projects, 1M). Growth $299 (10 projects, 10M)
Watch-outIt will not 403 anyone. The Free 100K cap is easier to blow than the $0 line suggests
|
| Fastly | The site already sits on Fastly and you want AI crawler policy in front of cache |
ControlEdge classify, then allow, block, rate-limit, or challenge. ContentGuard inspects in front of cache
PriceQuote. Bot Management and AI Bot Management are contact-sales on the public pricing page. Billed per million requests processed
Watch-outYou need a paid Fastly account and a delivery product. Essentials does not include Bot Management. ContentGuard can raise the bot bill
|
| Cloudflare | Per-crawler allow / block / charge on a zone you already run there |
ControlAI Crawl Control on every plan. Custom 403 / 402 responses on paid plans. Pay per crawl is closed beta
PriceCore controls run on existing plans. No public add-on list rate. Custom block responses need a paid plan
Watch-outFree-plan detection is user-agent based. Pay per crawl is invite-only. Do not put revenue from it in a 2026 plan
|
| Radware | Publishers who need the same crawler policy on article URLs, the subscription API, and the app |
ControlIntent, fingerprinting, and a Crypto Challenge. Mobile SDKs attest the device. SaaS or a virtual appliance
PriceAWS: $1,130.00 / month up to 50 million requests. 30-day $0 trial. Higher volume is contact-us
Watch-outA full estate quote is a separate conversation. Reviewers still report occasional blocks on friendly crawlers after go-live
|
| Akamai Bot Manager | Publishers already on Akamai who need edge scoring, then Watch / Challenge / mitigate on article URLs |
ControlScore 0-100 from the first request. Cautious / Strict / Aggressive responses on the same property
PriceQuote-only. App & API Protector includes a lighter bot view. Premier is the transactional add-on
Watch-outDo not use third-party monthly figures. Included-with-WAF bots are not Premier
|
| Imperva Advanced Bot Protection | Publishers already on Imperva who need measure-then-enforce on scrapers and logins, not a second CDN |
Control700-plus dimensions, then custom responses on web, API, and mobile
PriceQuote-only. Free trial on the product page
Watch-outA WAF or DDoS plan alone is the wrong product for this job
|
Known Agents
Known Agents is not an edge enforcer. It is analytics for the machines hitting your site: which named agent, which page, what it did, and whether a human later arrived from ChatGPT, Claude, Gemini, or Perplexity. The Agent Analytics page tracks 2,000-plus crawlers, scrapers, and AI agents across 15-plus categories. An auto-updating robots.txt adds new bots as they show up. A verification API returns agent token, type, and operator. Founded in New York by Gavin King. Freemium, self-serve, no credit card on the free tier.
This is not an edge enforcer. Pair it with Cloudflare or Fastly when you are ready to 403 a category. Teams over-block. A $0 project that names GPTBot versus a spoofed Chrome string is cheaper than a month of guessing from CDN logs.
Best for: Analytics first, not enforcement. Publishers, docs-heavy SaaS, and SEO teams that need to see which agents cite or convert before they write a deny rule.
What you get:
- Per-agent timelines, pages, spike alerts, and failure analysis (errors, redirects, blocks).
- LLM referral attribution from the major chat platforms.
- Automatic robots.txt, with WordPress rule enforcement on paid tiers.
- Agent Identification API and MCP access so an assistant can query the project.
Why we like it: You can see who is citing you before you cut the referral. The public grid is the only self-serve dollar table on this shortlist besides the Radware Marketplace floor.
Limits:
- It will not stop a crawler at the edge. You still need a CDN or WAF policy.
- Public third-party review volume is thin. Run it on one property first.
- The free plan hard-caps events. Paid plans email you before they cut you off. The plan grid does not say whether unused events roll over.
Price: First-party Known Agents pricing, monthly: Free $0 (1 project, 100K events), Startup $29 (3 projects, 1M events), Growth $299 (10 projects, 10M events), Enterprise custom. Yearly billing is on the same page. An event is one API call (visit, identification, or other). The FAQ says start with roughly twice your monthly human visit count, because agents are about half of traffic. The Free 100K cap is easier to blow than the $0 line suggests.
Fastly
Fastly AI Bot Management is the isolate-and-block action on Fastly’s edge: classify the request, then allow, block, rate-limit, or challenge. ContentGuard inspects in front of cache, which is the lever when AI crawlers are hitting your most-cached URLs. Post-cache inspection rides the Next-Gen WAF on cache-miss traffic bound for origin. Client challenges are scoped per fully qualified domain name. Embedded Dynamic Challenges shipped in Q1 2026 so login and checkout do not bounce to an interstitial.
You need a paid Fastly account and a delivery product. Bot Management is in the Security Subscription tiers except Essentials. Billing is per million requests processed, separate from delivery and from the Next-Gen WAF. ContentGuard can raise that bot bill because it sees cached hits too. The public Fastly pricing page lists both Bot Management and AI Bot Management as contact-sales. You are buying Fastly. This is not a bolt-on for a Cloudflare or Akamai shop.
Best for: Teams already on Fastly CDN or Next-Gen WAF that want AI crawler policy at the edge, including in front of cache.
What you get:
- Edge classification from request signals, behavior, and client attributes.
- ContentGuard pre-cache inspection for scrapers hitting hot objects.
- Post-cache mode with Next-Gen WAF on origin-bound misses.
- Per-FQDN client challenges, including Embedded Dynamic Challenges.
Why we like it: Origin load drops the day a noisy crawler is stopped in front of cache. That is a bandwidth line item, not a robots.txt hope.
Limits:
- No public list rate. Usage-based, and reviewers call it expensive.
- Post-cache needs Next-Gen WAF. Essentials does not include Bot Management. Scope the bundle before the POC looks cheap.
- You are buying Fastly. This is not a bolt-on for another CDN.
Price: Quote. Charged on request volume per month. Bot Management and AI Bot Management are both “Contact sales for pricing” on fastly.com/pricing. Set a spend alert. Contact sales or your CSM.
Cloudflare
Cloudflare AI Crawl Control is per-crawler policy on a Cloudflare zone: allow or block, watch robots.txt compliance, return 403 or 402 with a licensing message on paid plans, and optionally charge. It deploys with no extra hostname setup on every plan. Custom block responses and pay per crawl need a paid plan. AI Audit was the preview name. It is AI Crawl Control now, generally available since August 2025. On the free plan, detection is user-agent based. Paid plans can use Bot Management detection IDs.
Pay per crawl is still closed beta. Cloudflare is merchant of record. The crawler sends payment intent headers or gets HTTP 402 with a price. On 16 Jun 2026 Cloudflare added URI-pattern exclusions (free homepage, charge the archive) and dynamic pricing via a crawler-price origin header or a Worker. Payment headers must be in the Web Bot Auth signature-input so a crawler cannot swap the price after signing. WAF or Bot Management blocks override charge. Web Bot Auth is the signed-identity path. Signed identity is not universal. Expect holdouts.
Best for: Publishers and brands already on Cloudflare who want audit-then-enforce per agent, and who may want a 402 experiment without writing a billing stack.
What you get:
- Per-crawler allow / block with request counts and trend charts, on every plan.
- Configurable 403 or 402 responses on paid plans.
- Pay per crawl (closed beta), Cloudflare as merchant of record, dynamic price headers as of 16 Jun 2026.
- Web Bot Auth cryptographic signatures for crawler identity.
Why we like it: You can start in observe, then deny training crawlers and leave search fetchers alone, from the same dashboard that already has the zone. No extra hostname setup.
Limits:
- Signed identity is not universal. Cloudflare has publicly accused crawlers of spoofing Chrome after a block.
- Pay per crawl is beta, invite-only. Do not put revenue from it in a 2026 plan.
- Exact plan fees for AI Crawl Control extras are not listed. Custom 402 responses require a paid plan.
Price: Core controls run on existing Cloudflare plans. Custom block responses (403 or 402 body and code) are available on paid plans. Pay per crawl needs beta access. No public add-on list rate.
Radware
Radware Bot Manager (now sold as Bot & Agent Management inside Cloud Application Protection) classifies automated traffic across web, API, and native mobile. Intent-based behavior analysis, device fingerprinting, collective bot intelligence, and a Crypto Challenge (proof of work, no CAPTCHA) for humans. Mobile SDKs do device attestation on iOS and Android and block emulators, modified apps, and tampered OS images. Deployment is SaaS or an on-prem virtual appliance.
For a publisher, that matters when the same article URLs, the subscription API, and the iOS app all see the same scraper. A CDN crawler toggle will not cover the app. The Crypto Challenge and search-crawler detection are how you keep Googlebot while you stop the archive scrape. When stuffing and scraping arrive on the same API as the app, a zone toggle is the wrong layer.
The AWS Marketplace listing is the only public enterprise rate on this shortlist besides Known Agents: a 30-day free trial, then a monthly request-volume dimension. Higher volume is contact-us. Contract term is 1 or 12 months. A full estate quote is a separate conversation. Virtual appliance is a separate conversation.
Best for: Publishers that need the same crawler policy on article URLs, the subscription API, and the mobile app, and that may require an appliance.
What you get:
- Allow, block, and rate-limit by intent, not just user-agent.
- Native mobile attestation. Web-only tools miss this path.
- Crypto Challenge instead of a CAPTCHA wall.
- Legitimate search-crawler detection so aggressive rules do not nuke SEO.
Why we like it: A publisher can keep Googlebot and stop an archive scrape on the website, the subscription API, and the app with one policy. A CDN toggle will not cover the app. The Marketplace floor is a number you can put in a budget line while the direct deal is still a quote.
Limits:
- Premium pricing and a tuning period. Reviewers still report occasional blocks on friendly crawlers after go-live.
- Reporting customization is a recurring complaint.
- The marketplace listing is a request-volume dimension. A full estate quote is a separate conversation.
Price: AWS Marketplace $1,130.00 / month up to 50 million requests, plus a 30-day $0 trial. Direct deals are quote. Virtual appliance is a separate conversation.
Akamai Bot Manager
Akamai Bot Manager scores each request from 0 (human) to 100 (bot) on the Akamai edge, starting with the first hit, then lets you set three response bands: Cautious (Watch), Strict (Challenge), and Aggressive (mitigate). Stealth actions sit next to block and allow, so you are not limited to a 403 that teaches the scraper to rotate. A known-bot directory and custom categories are how a search fetcher or a partner crawler gets through without opening the rest of the archive.
For a publisher already on Akamai delivery, that is the same property that already terminates the article request. You can Watch GPTBot, mitigate a spoofed Chrome scrape, and serve cached HTML to a noisy crawler so origin stays up during a traffic spike. Bot & Agent Control also names Content Protector for scrapers on content URLs and AI Brand Presence for how AI discovers and uses your pages. App & API Protector includes a lighter bot view. Bot Manager Premier is the behavioral add-on for transactional URLs. On a news or docs site, Premier matters if the hole is subscribe and login. The article archive is Bot Manager, not Premier.
Best for: Publishers already on Akamai who need edge scoring, then Watch / Challenge / mitigate on article URLs, without standing up a second CDN.
What you get:
- A 0-100 bot score from the first request, with Cautious / Strict / Aggressive responses you can tune.
- A known-bot directory plus custom categories for partner and search crawlers.
- Stealth responses beyond block and allow, and the option to serve cached content to bots instead of live origin.
- The same detections on mobile apps. Content Protector and Account Protector sit as adjacent products on the same hub.
Why we like it: You can treat a training crawler and a search fetcher as different policies on the Akamai property you already run. Watch one. Mitigate the other. That is a publisher control, not a general bot dashboard.
Limits:
- Quote-only. Do not use third-party $3,000 to $4,000 / month figures.
- The lighter bot view inside App & API Protector is not Premier, and Premier is not the article-scrape product.
- This is an Akamai-edge product. It is not a bolt-on for a Cloudflare or Fastly shop.
Price: Quote-only. First-party pages are Bot Manager and Bot & Agent Control. No public list rate.
Imperva Advanced Bot Protection
Imperva Advanced Bot Protection is the dedicated bot layer on Imperva WAAP: measure, then enforce, on websites, APIs, and mobile apps. Detection combines client interrogation, behavior, machine learning, connection traits, and threat intel across 700-plus dimensions. Dashboards and explainable detections are the measure step. Out-of-the-box and custom responses cover OWASP automated threats. A human analyst service tunes policy after go-live. Quote-only. A free trial sits on the product page.
The Bad Bot Report is their research. The product is Advanced Bot Protection. For a publisher already on Imperva, that is the measure-then-enforce layer for scrapers on article URLs and abuse on subscribe or login, on the same edge that already terminates the request. It is not a robots.txt switch and not a reason to migrate onto a second CDN. First-party language says legitimate users see a CAPTCHA on 99.999% fewer requests when CAPTCHA is the chosen action. A WAF or DDoS plan alone is the wrong product if the job is the archive scrape.
Best for: Publishers already on Imperva who need to see which scrapers hit article URLs, then enforce without putting a challenge on every reader.
What you get:
- Measure first: 700-plus dimensions, real-time dashboards, custom reports, and production policy testing.
- Enforce on web, API, and mobile, including OWASP automated threats and custom responses.
- A path that can mitigate without a CAPTCHA for most readers when that is the chosen action.
- A human analyst service for tuning after you turn the policy on.
Why we like it: You can name the scraper on an article path, then enforce on the Imperva edge you already run, without a second CDN and without a CAPTCHA wall on every session. That is the publisher job. Credential stuffing is in scope. It is not the reason to buy this row if the hole is only AI crawlers on a Cloudflare zone.
Limits:
- Quote-only. No public list rate.
- It is a WAAP add-on. WAF or DDoS alone will not do this job.
- This is not a self-serve crawler dashboard. If you need a public $29 analytics grid first, start with Known Agents.
Price: Quote-only. Trial and demo on imperva.com/products/advanced-bot-protection.
Do you need to measure first, and does the deny sit on an edge you already run?
This grid plots two questions. Across is the job: measure and name the bot on the left, enforce (403 / 402 / challenge) on the right. Up is where the control sits: on a CDN you already run at the top, a separate product at the bottom.
ContentGuard, AI Crawl Control, or Bot Manager
Web, API, mobile, or a WAAP add-onPlacement is from first-party product language: an analytics project that does not sit on the edge, Bot Management on a Fastly delivery product, AI Crawl Control on a Cloudflare zone, Bot Manager on an Akamai property, and a full-stack or WAAP bot product when the same policy has to cover more than a CDN toggle. Axes are jobs, not points.
Pricing and usage costs compared
| Service | Published rate | What a spike costs |
|---|---|---|
| Known Agents | Free $0 / 100K events. Startup $29 / 1M. Growth $299 / 10M | Free hard-caps. Paid plans email you before they cut you off. An event is one API call. Unused events may not roll over - the page does not say |
| Fastly | Quote. Per million requests processed, separate from delivery | ContentGuard sees cached hits, so the bot line can rise when origin load falls. Essentials does not include Bot Management |
| Cloudflare | Included on existing plans. No add-on list rate | Custom 402 / 403 responses need a paid plan. Pay per crawl is beta and is not a 2026 revenue line |
| Radware | $1,130.00 / month up to 50 million requests. 30-day $0 trial | Above 50 million is contact-us. Appliance and a full estate quote are separate conversations |
| Akamai Bot Manager | Quote-only | Quote-only. Premier is the transactional add-on, not the article-scrape product. Do not use third-party monthly figures |
| Imperva Advanced Bot Protection | Quote-only | WAAP add-on. WAF or DDoS alone is the wrong product. Trial is on the product page |
What we left out
They sit next door because they lack a named analytics or edge product for AI crawlers, a public or quoted price, or a control that sits where the request actually dies.
- robots.txt alone - A directive, not a control plane. Well-behaved GPTBot will honor it. The residential-proxy swarm that spoofs Chrome will not.
- A second CDN you do not already run - Switching edges to buy a crawler toggle is a migration, not a bot product.
Questions before you 403 a crawler
Put these three in the SOW. If they will not, you are still buying a global deny you have not priced.
- Do you know which agents send humans back? If not, start with analytics. A 7% visit drop after a robots.txt GenAI block is a pricing decision, not a security win.
- Where should the deny land? On the edge that already terminates the request, in front of cache if scrapers hit hot objects, or on a full-stack bot product if the same policy has to cover web, API, and mobile. Origin-only rewrite rules will not save the bandwidth bill.
- Are you charging or blocking? A 402 path is the only serious public experiment. It is still beta. Everyone else is allow / block / challenge. Do not invent a licensing desk you cannot operate.
Which crawler control should you pick
Need a name on the bot before you write a rule: Known Agents, Free $0 then $29 / $299. Already on Fastly: ContentGuard, quote, and watch the pre-cache bill. Already on Cloudflare: AI Crawl Control on the zone you have, paid plan if you want a custom 402, and do not budget pay-per-crawl revenue. Same policy on article URLs, the subscription API, and the app: Radware, $1,130.00 / month up to 50 million requests on the Marketplace floor. Already on Akamai: Bot Manager, quote, Premier only if the hole is subscribe and login. Already on Imperva: Advanced Bot Protection, quote, and start in measure on article URLs. Then put that page next to a week of real agent traffic, not a user-agent guess.
Frequently asked questions
Is a robots.txt deny enough to stop AI crawlers?
No. Well-behaved training bots will honor it. Spoofed Chrome on a residential proxy will not. A global deny also cuts referrals you have not priced. Measure who cites you, then enforce on the edge that already terminates the request.
Can we stay on Known Agents Free and 403 later?
Known Agents will not 403 anyone. Free is $0 with 1 project and 100K events, hard-capped. Pair it with Cloudflare or Fastly when you are ready to deny a category. The plan grid does not say whether unused events roll over if you are modeling Growth as a buffer.
Should we put pay-per-crawl revenue in a 2026 plan?
No. Cloudflare’s 402 path is closed beta, invite-only, with Cloudflare as merchant of record. Custom 403 / 402 responses need a paid plan. Everyone else on this list is allow / block / challenge. Do not staff a licensing desk on a closed beta.
