Stack Overflow blocks nearly 900 million AI scraping requests per week. That number comes directly from Jody Bailey, Chief Product and Technology Officer at Stack Overflow. It is an operational figure from a platform that has been managing the bot problem at production scale long enough to watch it transform from a manageable nuisance into a structural threat to the economics of running a public platform. And the reason the number is that large is not that Stack Overflow's defenses are weak. It is that the thing they are defending against has fundamentally changed, and most of the mental models, the metrics, the tooling, and the verification frameworks that the industry built to handle the previous generation of bot traffic are being systematically defeated by the current one.
The old bot was a script. It broke flows, hammered endpoints, triggered obvious error patterns, and left traces in dashboards that any engineer monitoring latency and error rates could see without digging deeper. You could catch it because it behaved like a machine. But the new bot is something categorically different. It is an AI system that has been trained on enough human behavior to mimic it convincingly, read the environment it is operating in and choose not to look abnormal, and distribute its activity across identities, sessions, and IP addresses in ways that defeat the threshold-based rules that traditional WAFs were designed to enforce. Latency tells you your system is under pressure. It does not tell you your system is being quietly exploited by something that has learned exactly how much pressure a system needs to show before defenders notice.
Monitoring the Wrong Thing
The detection gap that AI-driven bots have opened up is not a gap in tooling sophistication. It is a gap in what the tooling was designed to look for. Bailey describes the shift his team made as a consequence of watching traditional monitoring fail against scrapers that had learned to behave like users. The focus moved from individual requests to behavior over time, from single-point metrics to request sequences, traffic shape at scale, and fingerprints that remain consistent even when IP addresses rotate. The question the monitoring infrastructure now tries to answer is not whether a given request looks unusual but whether the identity behind a sequence of requests is doing something that no real user would do over the same period. For practitioners operating at scale, that shift in framing is not optional. It is the foundational change that makes everything else in the detection architecture possible.
What that shift looks like at the architectural level is something Diana Kelley, Chief Information Security Officer at Noma Security, addresses with precision. Traditional telemetry, latency, throughput, and error rates, remains useful for tracking general service performance but cannot reliably detect sophisticated automated traffic because modern bots can emulate human timing and interaction patterns convincingly enough that classic performance signals stay within expected ranges while abuse is actively occurring.
"API monitoring and threat detection have evolved toward behavioral baselining, contextual correlation, and intent-aware analysis," Kelley explains, "rather than relying solely on static thresholds." The practical implication is that best practices now require combining traditional telemetry with session context, sequence patterns, velocity anomalies, client and device signals, and threat signatures to build detection precision that no single signal type can achieve alone. Building that layered picture is not a tooling upgrade. It is a rearchitecting of what the monitoring stack is asked to see.
The specific failure mode that makes intent-aware analysis necessary rather than optional is one that Kelley identifies with particular clarity. Agentic bots do not just make unusual requests. They chain individually legitimate requests into workflows that are abusive in aggregate. A single API call to fetch a user profile looks fine. A sequence of API calls that fetches ten thousand user profiles over six hours, paced to stay within rate limits at the individual endpoint level, is business logic exploitation that a request-level analysis will never surface. "Effective API analytics aim to detect abuse by correlating activity across identities, tokens, devices, and time," she argues, "using sequence and path analysis, unusual call combinations, velocity and retry patterns, and deviations from typical user, app, or client behavior." The detection architecture has to operate at the session and journey level rather than the request level to see what these systems are actually doing. Teams that have not made that transition are measuring the wrong thing and will continue to miss the threat until the economics of ignoring it become impossible to absorb.
Bills Before The Alerts
The economic argument for taking agentic bot traffic seriously is often framed around bandwidth and egress, and those costs are real. But understanding the true scale requires moving past the aggregate figures and into what is actually driving them. Saoud Khalifah, Co-Founder, CTO, and CEO of Ciphero, puts numbers behind the problem that make the conversation concrete rather than abstract. According to Khalifah, AI crawler traffic has surged 300 percent year over year, and over 80 percent of AI bot activity is crawlers looking for training data rather than bots executing transactions or performing account takeovers. The infrastructure carrying that traffic is being subsidized by the organizations whose platforms it is extracting from, without attribution, without compensation, and without the engagement signals that would make the traffic valuable to the platform generating the content. That subsidy is not a rounding error. At the scale Khalifah describes, it is a structural cost that compounds with every percentage point of crawler growth.
Collin Hogue-Spears, Senior Director of Product Management at Black Duck Software, makes the cost argument with a specificity that reframes where the real expense sits. The damage is not primarily in egress. It is in compute, and specifically in the expensive endpoints that sophisticated bots identify and exploit precisely because they are expensive. Search, recommendations, export workflows, media transforms, and API calls that trigger fan-out into multiple internal services are all endpoints where a single request can cause heavy downstream work while looking like normal behavior at the surface level.
"A sophisticated bot will find the expensive paths and keep using them while staying within what looks like normal behavior," Hogue-Spears notes, and the cloud bill arrives before the alert because the monitoring infrastructure is watching for volume anomalies rather than cost-per-identity signals. The actionable shift Hogue-Spears points to is tracking spend differently. "It is no longer enough to watch total traffic volume," he argues. "You have to understand cost per identity and cost per journey. When a single user is consuming disproportionate compute and data, that is not engagement. It is a signal that something is off." Tagging automated traffic at the edge and piping those tags into cost allocation is the operational mechanism that makes that analysis possible, but it requires a CDN that can distinguish crawler from customer and a cost allocation model that tracks spend at the identity level rather than the aggregate.
The economic argument becomes harder to ignore when the identity of the extractor shifts from a commercial crawler to a state-affiliated one. Open technical platforms, Stack Overflow included, sit at the intersection of software development and national security in ways that are rarely made explicit. The knowledge they hold about infrastructure patterns, vulnerability research, and engineering practice is strategically valuable, and large-scale automated extraction of that knowledge at a platform's expense is not categorically different from other forms of state-sponsored intelligence collection. Governments have not yet suggested coherent policy frameworks for this, which means the cost of defending against it falls entirely on the platforms and, by extension, on the engineers and organisations that depend on them.
CAPTCHA Already Lost
CAPTCHA is losing the fight, and the practitioners who have examined the current generation of AI-driven bots most carefully are consistent on why. The problem is not that CAPTCHAs have become easier to solve. It is that the assumption underlying CAPTCHA, that the challenge of solving a visual or cognitive puzzle is meaningfully harder for an automated system than for a human, no longer holds against multimodal models that can process images, read distorted text, and navigate click-based challenges faster and more reliably than a frustrated user on a mobile device. Khalifah adds a dimension that the technical discussion often misses. Even setting aside model capabilities, real people are paid to solve CAPTCHAs at scale to facilitate account creation for automated systems, which means the human-verification step is already being satisfied by humans in the service of automation. The implication is not that CAPTCHA needs to be improved. It is that the premise CAPTCHA rests on has been invalidated, and the industry needs a different question to answer.
The reframe Hogue-Spears offers is blunt. "The question of whether this is a human is already obsolete," he argues. "The scalable question is whether this identity is authorized for this specific action right now." That shift in framing requires a different architecture, and he is specific about what it requires in practice. Phishing-resistant authentication for human users. Cryptographic workload identity for machine clients, replacing static API keys that create attribution gaps and accumulate over time as credentials that nobody specifically owns or monitors. Step-up friction reserved for anomalous sequences rather than applied uniformly at every interaction point.
"If your non-human identities still use static API keys, you have lost attribution and you will pay for it in fraud and infrastructure spend before you see it in your SIEM," Hogue-Spears warns. Organizations that have not made the transition to short-lived tokens and signed requests for sensitive API surfaces are operating on a trust model that the current generation of automated clients was specifically designed to exploit.
Bailey's approach at Stack Overflow operationalizes the trust-over-time model at platform scale. Real users are consistent across sessions. Their behavior patterns, their fingerprints, their pacing, and their navigation paths stay stable in ways that automated systems impersonating users at scale cannot maintain without sacrificing the operational efficiency that makes large-scale scraping economical. The monitoring infrastructure watches for identities that do not behave like a person over time, even when each individual request looks clean, and applies targeted controls like stricter rate limiting or edge-level mitigations when the pattern becomes clear.
And critically, Bailey's team at Stack Overflow has been deliberate about not blocking or penalizing legitimate developers who automate interactions with the platform, because the line between a malicious scraper and a developer building a legitimate integration is drawn by intent and pattern rather than by the presence of automation itself.
Where This Goes Next
The next generation of verification mechanisms will not look like the current one, and the direction the field is moving reflects a fundamental shift in what the problem actually requires. Akhil Verghese, Founder and CEO of Krazimo, draws a distinction that most coverage of the post-CAPTCHA verification landscape overlooks. Cursor tracking, the measurement of mouse movement patterns as a human signal, is already being faked well enough to defeat the detection systems built around it. The verification mechanisms that will be harder to defeat are those that require a physical property rather than a behavioral one, something closer to biometric confirmation through a camera or a device-level attestation from a trusted browser that the platform can verify without seeing the underlying data. The shift from behavioral signals to physical ones is significant because it changes what an attacker has to replicate, moving from patterns that can be learned and mimicked to properties that require genuine physical presence.
The computational cost barrier is another direction Verghese identifies as viable at a different layer of the problem. Making bot interaction passively expensive enough to disincentivize crawling at scale, through background computation requirements that any browser can handle without the user noticing but that become prohibitively costly when run across thousands of concurrent bot sessions, applies economic friction to the automation problem rather than trying to detect and block it. The proof-of-work model at the bot-defense layer shifts the cost of scale back onto the attacker rather than the defender.
Khalifah points to a broader architectural shift that both of these directions are part of. The conversation about verifying human identity is moving away from challenge-based barriers, which are fundamentally puzzles that can be solved, and toward identity primitives that are harder to replicate at scale. Worldcoin's approach to proof-of-personhood, whatever one thinks of its specific implementation, is a signal of where the industry's thinking is going because the bot problem is forcing a reckoning with what it actually means to verify that an identity corresponds to a unique human being rather than an automated system pretending to be one. That is a harder problem than solving a distorted image, and it is the problem that the current generation of agentic bots has made unavoidable.
The organizations that are building the right architecture for this moment are the ones that have stopped trying to answer the question the old bot traffic posed and started building for the question the new traffic is asking. The old question was whether a request came from a human. The new question is whether the identity behind a sequence of requests is authorized, consistent, and trustworthy across the full context of what it is doing. Answering that question well requires a detection architecture that operates across time and identity rather than at the request level, a verification model that applies friction at the moments that actually matter rather than uniformly at every entry point, and an economic model that makes the cost of large-scale automation fall on the attacker rather than the platform.
