The Internet: Rebuilt for AI

Traditional cloud infrastructure was built for human interaction patterns – consistent searching, clicking, scrolling, and streaming. AI agents, however, operate with vastly different demands, generating intense bursts of activity, such as launching numerous sub-agents to query databases, scour documents, and invoke APIs within seconds, only to vanish just as rapidly.

Recognizing this shift, Amazon is overhauling a critical component of its cloud infrastructure. AWS recently unveiled the latest iteration of OpenSearch Serverless, a comprehensive, fully managed search and vector database solution designed specifically to handle agentic workloads. This system, which facilitates large-scale information storage and retrieval, boasts the ability to instantly scale up in response to agent-initiated tasks and seamlessly scale down to zero when inactive.

This introduction underscores a burgeoning understanding within the technology sector: the foundational infrastructure originally conceived for a human-centric internet is becoming less suitable for an environment increasingly dominated by autonomous agents.

Although AI agents currently constitute a comparatively minor segment of overall internet activity, machine-generated traffic is already substantial and projected for significant expansion. Cloudflare reported that bots were responsible for 31% of all HTTP traffic over the past six months, with AI crawlers, search engines, and assistants contributing approximately a quarter of those bot requests.

"Non-human traffic is anticipated to surpass human traffic sometime within the first half of 2027," stated Li Yi Ohlsen, a senior product manager at Cloudflare, in an interview with TechCrunch.

During its recent I/O developer conference, Google announced that users would soon be able to delegate various tasks to AI systems, including purchase research, travel booking, web browsing, and application interaction. However, the impact extends beyond consumer-oriented AI agents, as enterprises are increasingly deploying these agents both internally and for customer-facing applications, thereby generating novel forms of machine-to-machine traffic behind the scenes.

Consequently, cloud providers and infrastructure firms are actively addressing the challenge of adapting systems originally designed for human users to an environment where agents autonomously retrieve information, invoke tools, and generate continuous machine-to-machine traffic.

This is precisely the scenario that AWS's new OpenSearch Serverless aims to address.

"The timing is clear," explained Tia White, General Manager for Amazon OpenSearch Service, to TechCrunch. "Agents are transitioning from experimental phases into full production, generating traffic patterns that existing infrastructure was simply not built to handle. They exhibit unpredictable spikes and equally sudden periods of idleness, and enterprises require a search solution that can keep pace without incurring costs for unused or idle compute resources."

The fundamental technical innovation in this latest generation lies in its decoupling of compute resources from storage. This architectural change enables compute to scale up in mere seconds to manage agent traffic bursts and then scale down entirely to zero, ensuring customers incur no costs during periods of agent inactivity.

White elaborated, "Previously, even with our earlier Serverless offering, a minimum of one operational instance was required because storage and compute were intrinsically linked. It wasn't possible to automatically provision compute resources precisely as needed, resulting in customers always having idle compute reserved for their workloads, irrespective of actual usage."

To illustrate, consider the prior model akin to consistently paying for a parking space, even when it remains empty. AWS's enhanced Serverless, by contrast, operates more like a metered parking spot, where payment aligns directly with usage.

Upon its release, OpenSearch Serverless will offer native integration with prominent AI development platforms such as Vercel and Kiro. This functionality empowers developers to deploy production-grade search and vector backends for their agents without the complexities of infrastructure management.

This transformative shift is evident across the broader cloud industry. Companies like Databricks and Snowflake are re-positioning their offerings as AI memory and retrieval systems for enterprise data. Microsoft, for its part, has introduced updates to Azure specifically engineered to manage AI agent bursts and facilitate shared memory between agents. Similarly, Cloudflare, echoing Amazon's strategic move, unveiled new infrastructure last month aimed at providing agents with persistent operating environments and immediate scalability.

As more organizations adopt and deploy AI agents, the imperative to redesign infrastructure around these unique machine-generated workloads will intensify, ultimately leading to more cost-effective and simpler deployment of agents at scale.