Automated traffic now outweighs people, and most security teams still treat it as a footnote
In 2024, automated software passed human beings as the largest source of web traffic for the first time in a decade. Imperva's 2025 Bad Bot Report put automated requests at roughly 51% of everything hitting websites, and the share attributed to AI crawlers and agents has climbed sharply since. That figure lands differently depending on where you sit. A publisher reads it as a revenue problem. A security architect should read it as an access-control problem that has quietly become the majority case: more than half of the requests reaching your applications are made by something that was never going to look at a login page, solve a CAPTCHA, or behave like a browser.
We spend a lot of our engagements untangling the consequences of that shift. A bank's public rate-quote endpoint getting scraped thousands of times an hour by a model-training crawler. A telecom's account portal absorbing credential-stuffing traffic that hides inside a wave of "AI assistant" user-agents. A logistics API bill that tripled because agents were re-fetching the same unchanged pages on a loop. None of these are exotic. They are the predictable result of pointing 2015-era bot controls at 2026-era traffic.
The good news is that the tooling has caught up in the last eighteen months, and the change is architectural rather than cosmetic. This is a guide to what actually works: classifying automated traffic by what it does, verifying machine identity with cryptography instead of guesswork, and enforcing policy at the edge before a request ever touches your origin.
Why bot traffic is a security concern, not just a bandwidth one
It is tempting to file crawler traffic under "cost of doing business on the internet." That framing hides four distinct risks that we see materialise in production.
The first is data exposure. Anything your web application renders to an anonymous request is now, in practice, training data or agent-retrievable content. Pricing, inventory, internal knowledge-base articles that were only ever "unlisted," personal data on profile pages, document metadata in search results. Scrapers do not respect the implicit assumption that low-traffic pages are private. If it returns a 200 to a bot, it is public.
The second is threat traffic wearing a friendly costume. Credential stuffing, inventory hoarding, gift-card cracking, and vulnerability scanning increasingly present themselves with plausible AI-agent user-agent strings, because defenders have been trained to wave those through. When "GPTBot" and "an attacker's scraper claiming to be GPTBot" are indistinguishable in your logs, your allowlist is doing the attacker's work for them.
The third is cost and availability. Cloudflare has reported that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have not changed since the last visit. That is pure waste landing on your origin, your database, and your egress bill, and it competes for the same capacity your paying users need. At the tail, aggressive crawlers become an unintentional denial-of-service.
The fourth is governance and audit. If you cannot say which automated parties accessed which content, you cannot answer a regulator, a data-protection request, or a contract clause about how your data may be used downstream. For anyone operating under ISO 27001, PCI-DSS, or a NIST CSF programme, "we don't log or classify machine access to our public estate" is a gap that shows up the moment someone asks.
The controls most teams are still relying on, and where each one breaks
Before getting to what works, it is worth being honest about the defaults, because most organisations are running some combination of these and assuming they are covered.
robots.txt is a request, not a rule. It is a text file that tells well-behaved crawlers what you would prefer they not index. It has no enforcement whatsoever. A crawler that ignores it faces no technical barrier, and the crawlers you most want to stop are exactly the ones that ignore it. Treat robots.txt as a courtesy signal for the polite, never as a control.
User-Agent filtering rests on a header the client chooses for itself. Any script can announce itself as Googlebot, as Chrome on macOS, or as your own internal service. Allowing or blocking on the User-Agent string alone means trusting an attacker to describe themselves accurately. It is the weakest identity signal on the request, and it has been for years.
IP allowlists made sense when crawlers ran from stable, published address ranges. They age badly. Legitimate agents now run from the same hyperscaler IP space as everyone else, published bot ranges change without notice, and residential-proxy networks give abusive traffic a fresh address on every request. Maintaining IP allowlists at scale becomes a full-time job that still leaves gaps.
CAPTCHAs and interstitials punish the wrong audience. They add friction for real users and assistive technology, they are increasingly solved by the automation you were trying to stop, and they do nothing for API traffic where there is no human to challenge. They have a place for narrow, high-risk flows, but they are not a traffic-governance strategy.
The common thread is that all four try to make a policy decision without a trustworthy answer to the most basic question: who, or what, is actually making this request?
Step one: classify automated traffic by behaviour
You cannot write sensible policy against a single bucket called "bots." The useful move, and the one the major edge providers have converged on, is to sort automated traffic by what it is trying to accomplish, because the business and security consequences differ enormously.
A practical taxonomy has four categories. Search indexing builds an index so an engine can answer questions later and, crucially, sends referral visitors back to you. Real-time agents act on a person's behalf during a live task, fetching a page while a human waits on the answer. Training crawlers absorb your content into a model, with no referral and no attribution flowing back. Transactional and data-collection bots perform checkout actions, price monitoring, or bulk extraction. A single crawler can do several of these at once, which is precisely why you want to record all of the behaviours it exhibits rather than pinning it to one label.
This classification is what turns "block all bots" into a decision you can actually defend. You almost certainly want the search indexer that drives traffic to you. You may want to allow, throttle, or price the training crawler depending on your content strategy. You want the credential-stuffing bot gone. Behaviour-based classification is available today in Cloudflare Bot Management, AWS WAF Bot Control, Akamai, and Fastly, and it is the foundation every later control builds on.
Step two: verify machine identity with cryptography
This is the shift that matters most, and it is the reason the last year has been more than incremental. The industry has agreed on a way for an automated client to prove who it is, rather than merely assert it.
The mechanism is Web Bot Auth, and it is built on HTTP Message Signatures, standardised as RFC 9421. The model is straightforward. A bot operator holds a private key and publishes the matching public key at a known, verifiable location. On each request, the operator signs the message with its private key. The edge validates that signature before the request proceeds. A valid signature replaces a spoofable claim with a verifiable one: this request genuinely came from the party that controls that key, and nothing else can forge it.
What makes this credible rather than theoretical is that it is not one vendor's proprietary scheme. It is an IETF effort, and adoption is already broad. Cloudflare has folded message signatures into its Verified Bots programme. Google is testing Web Bot Auth for agent verification against its own crawling infrastructure. Akamai and AWS WAF Bot Control both support authenticating legitimate agent traffic through the same signature approach. When Google, Cloudflare, Akamai, and AWS align on a wire format, it stops being an experiment and starts being the direction of travel.
For a security team, the design principle here is one you already apply to human identity: verify explicitly, and stop trusting attributes the client can set for free. Cryptographic bot verification is zero trust extended to non-human identity. The User-Agent string becomes a hint; the signature becomes the decision.
Step three: enforce identity, permission, and action at the edge
Classification tells you what a request is doing. Verification tells you who is making it. Enforcement is where the two become a policy, and the right place to run that policy is the edge, before the request reaches your origin.
The pattern is a short pipeline applied to every automated request. Establish identity through the signature. Check permission by looking up what this verified party is allowed to do against this content, drawing on the behaviour classification and the preferences you have set. Then take an action: allow, block, throttle, serve a reduced response, or charge. Consolidating authentication, authorisation, and metering at a single gateway is the same architecture that shows up in service meshes and API gateways, applied now to the open web. The origin only ever sees requests that have already cleared the gate, and your application servers stop spending cycles on traffic you were never going to serve.
Two enforcement tools deserve specific mention because they cover the traffic that verification alone cannot.
Rate limiting is your answer for traffic that stays anonymous. Not every legitimate source can or will identify itself, and privacy-sensitive clients are a real category. Per-client rate limits, applied by fingerprint or token rather than raw IP, let you keep unidentified traffic within bounds without an outright block. This is also your first line against the re-fetch waste and the low-and-slow scraping that never trips a signature check.
Deception and tarpitting handle the actively hostile. For crawlers that ignore every signal and hammer your estate, edge providers now offer maze-style responses that feed abusive scrapers an endless supply of generated, low-value pages, raising their cost without touching real users. Used carefully, it turns your most aggressive scrapers into their own bottleneck.
The emerging payment layer, and why it is really an access-control primitive
There is a newer development worth understanding even though it is still early: paying for a request inline over HTTP. The HTTP status code 402, Payment Required, has been reserved in the standard for decades and largely unused. Protocols such as x402 revive it so that a server can answer a priced request with a 402, a small payload stating the price and where to pay, and then serve the resource once the client re-sends with proof of payment attached.
Set aside the commercial framing for a moment. For a security architect, the interesting part is that payment becomes a credential. A caller with no prior relationship, no account, and no API key can be granted access purely by presenting valid proof, which is a clean way to gate anonymous machine traffic without the signup overhead that per-seat licensing always demanded. It is another verifiable claim you can require at the edge, sitting alongside the signature. It is genuinely experimental today, the gateways are largely on waitlists, and adoption depends on clients built to honour a 402, so treat it as something to track rather than deploy. But the underlying idea, a machine-checkable access token that travels with the request, fits the same edge-enforcement model as everything above.
Comparing the mechanisms side by side
ControlWhat it actually stopsWhere it breaks
robots.txt
Polite crawlers that opt in to your preferences
Zero enforcement; ignored by exactly the bots you care about
User-Agent filtering
Casual, lazily-labelled scripts
Header is client-controlled and trivially spoofed
IP allowlists
Bots from stable, published ranges
Cloud IP churn and residential proxies defeat it; high maintenance
CAPTCHA / interstitials
Some interactive automation on human-facing flows
Hurts real users and accessibility; useless for APIs; increasingly solved
Behaviour classification (WAF bot management)
Mislabelled and clearly malicious automated traffic
Classification confidence, not identity proof, on its own
Web Bot Auth (RFC 9421 signatures)
Impersonation of legitimate bots; forged identity
Only covers traffic willing and able to identify itself
Per-client rate limiting
Volumetric scraping, re-fetch waste, low-and-slow abuse
Does not distinguish good from bad, only fast from slow
Inline payment (x402 / HTTP 402)
Free anonymous access to priced resources
Early-stage; depends on client and ecosystem adoption
No single row is a strategy. The working posture layers them: classify everything, verify what can be verified, price or throttle what cannot, and log all of it.
An implementation checklist for security teams
- Inventory your automated traffic first. Turn on behaviour classification at your edge or WAF in monitor mode and measure what is actually reaching you, by category, before you write a single block rule. Most teams are surprised by the split.
- Stop making allow/block decisions on User-Agent alone. Where you currently allowlist named bots by their string, move those decisions to verified identity as the crawlers you care about adopt Web Bot Auth.
- Enable cryptographic bot verification through your edge provider's verified-bots or message-signature programme, and prefer verified traffic in your policy over merely self-declared traffic.
- Set per-client rate limits keyed on something better than raw IP, so unidentified and privacy-sensitive traffic is bounded rather than blocked outright.
- Define content tiers. Decide explicitly which parts of your estate may be indexed, which may be used for training, and which should never be served to automation at all, then enforce that at the edge rather than hoping robots.txt is honoured.
- Log machine access as a first-class audit stream. Capture verified identity, behaviour class, decision, and outcome, and retain it to satisfy ISO 27001 access-monitoring, NIST CSF detection, and PCI-DSS web-application logging requirements.
- Fold this into your non-human identity programme. Signed crawlers are one instance of the broader machine-identity problem your service accounts, workload identities, and internal agents also belong to. Govern them together.
- Re-test after every change. Bot posture drifts as crawlers evolve; a control that worked last quarter can be stale this one. Treat it as a monitored, iterated control, not a one-time deployment.
Where this connects to compliance
None of the above is a side quest from your existing framework work; it maps directly onto it. Access monitoring and the control of who reaches your information assets sit at the centre of ISO 27001's Annex A controls, and machine callers are information consumers like any other. NIST CSF's Detect function assumes you can see and characterise the traffic on your estate, which you cannot do if half of it is an unclassified blur. PCI-DSS expects logging and protection of internet-facing applications, and CIS Benchmarks push the same hardening discipline down to the services doing the serving. Building verifiable identity and behaviour classification into your edge gives you the evidence these frameworks ask for, as a by-product of a control you needed anyway.
How Aydahwa Enterprise can help
We work with banks, telecoms, and critical-infrastructure operators who cannot treat "more than half our traffic is machines" as someone else's problem. Our team brings hands-on architecture experience across WAF and bot-management deployment, edge and cloud security, and zero-trust design, backed by ISO 27001, PCI-DSS, SOC 2, NIST CSF, and CIS Benchmark practice and Microsoft Cybersecurity Architect Expert certification.
A typical engagement starts with measuring what is actually hitting your applications, moves through designing and deploying a layered control set, verification, classification, rate limiting, and audit logging, and ends with the compliance mapping that proves it. If you want to know where you stand before committing to a project, our cyber security self-assessment and cybersecurity readiness checklist are a fast, practical starting point. For the underlying capabilities, see our cybersecurity services and cloud security and migration work, and our managed IT support for ongoing operation. When you are ready to talk specifics, get in touch and we will scope it against your estate.
