Design Proposal: Native AI Support in Envoy

# Goal
We propose treating AI protocols as the distinct and AI-native layer in the Envoy, establishing native pathways to support the AI protocols rather than squeezing them into the HTTP filter chain.

Ultimately, this aims to make Envoy the AI proxy.

# Background
Large Language Models (LLMs) and the emergence of AI Agents have created a new set of requirements for networking proxies. While traditional proxies handle HTTP traffic, AI-native workloads require handling real-time streaming, large payload and diverse protocols (like OpenAI API, MCP, and A2A).

As the AI world rapidly evolves, [networking protocols can be treated as the next layers in the OSI model](https://www.linkedin.com/posts/annaberenberg_osi-model-of-7-networking-layers-seems-to-activity-7417306877128105984-WAhb/).

# Motivation
Envoy's hallmark is extensibility and flexibility. However, its current HTTP-centric architecture presents significant hurdles for effectively handling the unique demands of AI workloads: diverge support with as a set of ad-hoc filters/parsers, increased CPU/memory costs due to repeated parsing and transcoding, unidiomatic filter implementations that struggle with HTTP layer complexity.

## The "Why": AI Layer vs. HTTP Filters
This gap between Envoy's potential and the current HTTP filters for AI has led to:

- **Performance Hits & Reliability Concern**: Significant CPU and memory overhead is incurred by repeated buffering, parsing, and transcoding of HTTP payloads within Envoy's already complex architecture..

- **Orchestration Gaps**: The lack of a universal AI orchestration protocol layer forces Envoy to rely on ad-hoc filters for diverse specifications (JSON-RPC, JSON, SSE, aws_eventstream etc.).

- **Complex Developer Journey**: The need for developers to manage Envoy's HTTP internal mechanics to interpret AI protocols creates a high barrier to entry that alienates typical AI developers and OSS contributors.

- **Fragmented Development Efforts**: AI features are being developed in a siloed manner across numerous bespoke HTTP filters. Subtle implementation difference or inconsistency with HTTP filter chain contract could easily lead to bugs and security risk.

- **Limited Native Support**: The HTTP-centric paradigm forces many OSS users to rely on multiple ext_proc filters to offload AI protocol handling to external processes. This approach shifts significant complexity to users and hinders the development of shared community solutions. Besides, the multi-ext_proc approach has encountered reliability challenges.

# Architecture
We propose an AI-first design to establish dedicated filter chains for AI protocols that makes it easier to add AI functionalities into the Envoy by abstracting away HTTP semantics, and providing a more ergonomic set of interfaces to access and manipulate AI protocols.

Envoy has a well-established pattern of protocol layer lifting: a terminal network filter (HttpConnectionManager) on the current layer (L4 transport) performs decoding to manage filters on the next layer (HTTP filters).

Following this pattern, a new HTTP filter — **AI Protocol Manager** — is introduced to handle the decoding process: parse various AI-specific protocols (such as OpenAI API, A2A, or MCP) to construct an internal representation of the information associated with an AI request. AI Protocol Manager also acts as a filter manager to invoke and manage a newly introduced AI filter chain. The AI representation (i.e. AI Request) is then fed through this purpose-built AI filter chain to execute AI-specific logics such as AI guardrails, budgets, rate limiting etc.

At the end of the AI Filter chain, the **AI Dispatch Filter** is designed to handle the encoding process: lowering the internal AI representation back to the HTTP semantics, optionally performing transcoding as a natural lifecycle step of this encoding process. Once encoded, the dispatch filter dispatches the data to the upstream through the router interface (chain-forward mode) or through AsyncClient instances (fanout mode).

# Benefit
This design:

- **Empower** AI native feature development such as model routing, token counting, budgeting, cost tracking. It allows developers to focus on AI-specific logic like tokenization, prefix-aware load balancing, and guardrail integration without deep expertise in Envoy's HTTP internals.,

- **Consolidates** AI protocol interpretation to avoid fragmented implementation that might introduce subtle bugs as typically seen in the protocol world and to strengthen security validation against new types of AI risks via a centralized mechanism (i.e., AI codec), analogous to the HTTP request smuggling prevention in HTTP codec.

- **Reduces** CPU and memory overhead by centralizing expensive buffering and parsing tasks, performing them only once per request and by avoiding overloading the metadata/filter state across filters in the chain.

- **Facilitates** smooth integration and transcoding with existing, broadly used services like REST APIs. Transcoding is now a natural lifecycle step where high-level AI messages are lowered to HTTP protocols—specifically JSON-RPC for Agentic-native systems or JSON REST for web services.

- **Brings** the innovation back to Envoy. It will benefit the broader ecosystem and Envoy community.

# Ecosystem Adoption
Ecosystem adoption remains a critical pillar of our strategy; we are deeply committed to fostering broad alignment and community integration. While much of the proposed AI-native architecture will function as internal implementation details abstracted from customers (i.e., zero customer impact), a primary customer-facing shift lies in our proposal for a novel AI-native callout as an alternative to ext_proc.

Even though ext_proc has been a successful story that enables Envoy/dataplane programmability and extensibility in industry (and will continue to play a crucial role), it also has encountered reliability challenges and created the gRPC service/ ecosystem lock-in. From the [2025 Postman State of the API](https://www.postman.com/state-of-api/2025/#introduction) report, HTTP REST remains the dominant winner at [93%](https://screenshot.googleplex.com/BxYXYb9jrPgBLLu) (vs gRPC at 14%) . Thus, It is time to build new generic AI Native callout protocol to embrace a broader ecosystem with dual stack support (HTTP REST and gRPC)

Beyond adopting more prevalent protocols, the AI-native callout empowers AI service providers by allowing them to focus on core AI business logic. By removing the need to manually handle technical complexities—like message parsing between AI-native structures and HTTP formats—this strategy significantly reduces operational overhead and adoption barriers.

# Status
This proposal remains in the early design and ideation phase, as detailed in the [preliminary design document](https://github.com/penguingao/envoy/blob/main/source/extensions/filters/http/ai_protocol_manager/DESIGN.md). Initial POC ([MCP AI Filter Chain](https://github.com/penguingao/envoy/pull/16) and [LLM/Inference AI filter chain](https://github.com/penguingao/envoy/pull/19)) have been developed to validate the architectural approach and demonstrate the core value of this native AI architecture.

We are actively iterating on the design doc to finalize the API first. At the same time, we highly value the community's perspective and welcome further insights, feedback, and ideas as we refine our strategy, evolve the design, and iron out the remaining details.

# Acknowledgment
This proposal incorporates discussion and input from @penguingao @yangfanud @botengyao @paul-r-gall @yanavlasov @agrawroh @wbpcode