feat(llm-gateway): support Anthropic Messages API end to end
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Is this related to a problem? Please describe.
The NVCF LLM routing stack does not accept the Anthropic Messages API. A
POST /v1/messages request returns 404, so Anthropic-compatible clients,
including Claude Code, need a protocol translation proxy in front of NVCF.
Translation is an incomplete workaround because Anthropic content blocks,
stream events, tool calls and results, thinking blocks, usage, errors, and beta
headers do not map losslessly to OpenAI Chat Completions.
The current main branch confirms that this is an endpoint allowlist gap at
multiple layers:
src/invocation-plane-services/llm-api-gateway/api/routes.goregisters Chat
Completions, Responses, and embeddings, but not/v1/messages.src/libraries/rust/stargate/crates/stargate/src/http_proxy.rshas explicit
proxy routes for/v1/chat/completions,/v1/responses, and
/v1/embeddings, but not/v1/messages.src/invocation-plane-services/vanity-gateway/gateway/h2.goand its config
model expose OpenAI endpoint sections but no Messages section.docs/user/llm-gateway.mddocuments the same three supported LLM paths.
The Gateway API HTTPRoute already uses a / path prefix. Stargate's lower
transport also preserves the HTTP method, path and query, body, and forwardable
headers across direct and reverse worker tunnels. The ingress and worker
transport therefore appear reusable; the missing work is primarily endpoint
registration, gateway behavior, protocol-safe forwarding, and release wiring.
Describe the solution you'd like
Add native, end-to-end support for POST /v1/messages. Preserve the Anthropic
wire contract instead of translating it through Chat Completions.
The implementation should cover these layers:
- LLM API Gateway:
- Register
/v1/messagesand extract<function-id>/<model-name>from the
requestmodelfield. - Apply the existing NVCF authorization, model URI allowlist, routing method,
rate limiting, session affinity, tracing, and metrics behavior. - Rewrite only the routed model prefix and proxy the Anthropic request and
response formats without converting content blocks or SSE events. - Preserve protocol headers such as
anthropic-versionand supported
anthropic-betavalues. Define an explicit forwarded-header policy and
strip the reported gateway-onlyextra-headersmetadata before the request
reaches the model backend. - Account for Anthropic
input_tokensandoutput_tokenswhen usage is
available, and safely release reservations when it is not.
- Register
- LLM request router (Stargate):
- Add
/v1/messagesto the HTTP proxy surface. - Preserve path/query, body, Anthropic headers, streaming response events,
upstream status, and error bodies over both direct and reverse tunnels. - Continue stripping hop-by-hop and internal routing/retry headers.
- Add
- Vanity Gateway:
- Add a Messages route and configuration section, including LLM Gateway
targets, model lookup, custom-header validation, request sizing, telemetry,
and shadowing behavior where supported.
- Add a Messages route and configuration section, including LLM Gateway
- Model and release configuration:
- Treat
/v1/messagesas a supportedllmConfig.urisvalue. - Publish compatible LLM API Gateway and Stargate images/charts and update the
self-managed stack catalog so both sides are upgraded together. - Document configuration and a public curl example.
- Treat
Acceptance criteria:
- A model that declares
/v1/messagescan serve both streaming and
non-streaming native Messages requests through the LLM invocation endpoint. - Requests for a model that does not declare
/v1/messagesfollow the current
model URI allowlist mode. - Basic text, system prompts, tool use and tool-result follow-up, thinking
content blocks, token limits, and Anthropic SSE events round-trip without an
OpenAI protocol conversion. anthropic-versionand supported beta headers reach the backend, while
gateway-onlyextra-headersmetadata and internal NVCF routing headers do
not leak into the model API.- Upstream 4xx/5xx status, Anthropic error bodies, response headers, client
cancellation, and long-running streams propagate correctly. - Unit and integration tests cover LLM API Gateway routing and accounting,
Stargate direct and reverse tunnels, Vanity Gateway configuration/routing,
and a Claude Code-compatible tool loop smoke test. - Metrics and spans use the stable
/v1/messagesendpoint label without
adding model names, request IDs, or other unbounded metric labels.
Describe alternatives you've considered
An external proxy can translate /v1/messages to /v1/chat/completions.
That is useful as a temporary workaround, but it adds another operational hop
and can lose Anthropic-specific behavior used by agent clients. A native
pass-through path provides a smaller and more faithful compatibility surface.
Additional context
- Anthropic Messages API: https://docs.anthropic.com/en/api/messages
- Anthropic LLM gateway guidance for Claude Code:
https://docs.anthropic.com/en/docs/claude-code/llm-gateway
No duplicate issue was found by searching this repository for /v1/messages,
Anthropic, Messages API, or Claude Code.
By submitting this issue, you agree to follow our
code of conduct and our
contributing guidelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/invocation-plane-services/llm-api-gateway/api/routes.go, src/libraries/rust/stargate/crates/stargate/src/http_proxy.rs, and src/invocation-plane-services/vanity-gateway/gateway/h2.go, then review docs/user/llm-gateway.md and the related configuration models. Trace existing Chat Completions routing, proxying, and release configuration before identifying corresponding tests. Done means native streaming and non-streaming /v1/messages requests pass through all named layers with headers, errors, accounting, and configuration behavior covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- api, backend, cloud, release
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100