AI traffic needs more than an API gateway

Google Cloud is arguing that Envoy can become an open traffic layer for agentic systems, extending its established role in cloud-native networking to model calls, Model Context Protocol tools and agent-to-agent communication. Conventional Layer 7 proxies understand requests, identities and services, but an agentic workflow may stream tokens for minutes, invoke several tools, call another agent and change direction after new evidence. Managing that traffic requires awareness of model semantics and task state while retaining the reliability controls operators already expect from production networking.

Envoy brings a mature operational foundation

Envoy is widely used as an edge and service proxy and already underpins Google Cloud load balancing and the Agent Substrate dataplane on Google Kubernetes Engine. That history matters because AI platforms should not rebuild connection management, retries, telemetry and policy enforcement from scratch. An Envoy-based layer could give teams a consistent place to authenticate requests, choose destinations, apply budgets and observe latency. Open infrastructure also reduces the risk that every model provider imposes a different gateway pattern. The challenge is adding AI awareness without turning a general proxy into an opaque, vendor-specific appliance.

Model routing becomes a policy decision

An agent may choose between models based on capability, cost, latency, region or data sensitivity. Routing cannot rely only on a URL when the same task may be served by different providers. Policies need to preserve user intent and explain why a request moved. A cheaper model may be appropriate for classification but not for a high-impact legal analysis. If a preferred endpoint fails, a fallback could violate residency or retention requirements. The traffic layer should therefore make routing criteria explicit, auditable and testable, with hard constraints separated from optimisation preferences.

Tokens change familiar traffic economics

AI systems consume variable input and output tokens, and long contexts can dominate cost before a response begins. A token-aware proxy can enforce limits, attribute usage and detect abnormal growth across a multi-agent run. It should avoid reducing every decision to token count. Cached context, reasoning modes and provider pricing complicate comparisons, while a short incorrect answer can cost more operationally than a longer reliable one. Useful controls combine token budgets with task outcomes, latency and quality evaluation. Operators need to see cost across the whole workflow, not only one model invocation.

MCP expands the security boundary

Routing MCP traffic means handling requests that can read files, query databases or change external systems. A network proxy can authenticate connections and enforce endpoint policy, but it cannot assume that tool arguments are safe because they arrived from an approved model. Prompt injection may persuade an agent to call a legitimate tool with harmful parameters. Deployments need schema validation, least-privilege identities and approval for consequential actions. The proxy should preserve the initiating user and agent identity across calls so downstream services can make their own authorisation decisions.

Streaming and retries require special care

Model responses and tool results are often streamed, making partial failure more complicated than a standard request. Retrying an inference call may duplicate cost; retrying a tool action may send two emails or create two records. Envoy policies must distinguish idempotent reads from state-changing operations and surface uncertainty when a connection drops after an action may have completed. Trace context should follow every model, tool and agent hop. Without end-to-end correlation, teams cannot reconstruct why an automated workflow reached an unexpected result.

Open standards can reduce platform fragmentation

Envoy, MCP and agent-to-agent protocols could form a portable layer between applications and model providers. Portability will depend on conformance rather than branding. Extensions should expose stable configuration, metrics and failure semantics, and providers should avoid hiding critical behaviour behind proprietary control planes. Enterprises may still use managed services, but open data-plane components provide leverage and an inspection point. They also let security teams deploy policy close to workloads without sending every tool invocation through a remote third party.

The proxy is necessary infrastructure, not a complete defence

Google’s proposal is persuasive because agentic systems need operational plumbing as much as capable models. Envoy can provide a dependable foundation for routing, observability and enforcement across heterogeneous services. It cannot determine whether an agent’s plan is sensible or whether retrieved content is truthful. Production stacks will need model evaluation, tool-level authorisation, human escalation and recovery alongside the proxy. The most successful design will keep those responsibilities distinct while preserving one coherent trace. That is how open networking infrastructure can make agent systems easier to operate without creating a false impression that traffic control alone makes autonomy safe. Operators should rehearse provider outages, partial streams and duplicated tool calls before production. Shared dashboards should connect network events to task outcomes, allowing security, platform and application teams to diagnose one run from the same evidence during urgent production incidents today.