NVIDIA / NVIDIA/nvcf

feat(llm-gateway): support Anthropic Messages API end to end

Open
#1,503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Is this related to a problem? Please describe.

The NVCF LLM routing stack does not accept the Anthropic Messages API. A
POST /v1/messages request returns 404, so Anthropic-compatible clients,
including Claude Code, need a protocol translation proxy in front of NVCF.
Translation is an incomplete workaround because Anthropic content blocks,
stream events, tool calls and results, thinking blocks, usage, errors, and beta
headers do not map losslessly to OpenAI Chat Completions.

The current main branch confirms that this is an endpoint allowlist gap at
multiple layers:

  • src/invocation-plane-services/llm-api-gateway/api/routes.go registers Chat
    Completions, Responses, and embeddings, but not /v1/messages.
  • src/libraries/rust/stargate/crates/stargate/src/http_proxy.rs has explicit
    proxy routes for /v1/chat/completions, /v1/responses, and
    /v1/embeddings, but not /v1/messages.
  • src/invocation-plane-services/vanity-gateway/gateway/h2.go and its config
    model expose OpenAI endpoint sections but no Messages section.
  • docs/user/llm-gateway.md documents the same three supported LLM paths.

The Gateway API HTTPRoute already uses a / path prefix. Stargate's lower
transport also preserves the HTTP method, path and query, body, and forwardable
headers across direct and reverse worker tunnels. The ingress and worker
transport therefore appear reusable; the missing work is primarily endpoint
registration, gateway behavior, protocol-safe forwarding, and release wiring.

Describe the solution you'd like

Add native, end-to-end support for POST /v1/messages. Preserve the Anthropic
wire contract instead of translating it through Chat Completions.

The implementation should cover these layers:

  • LLM API Gateway:
    • Register /v1/messages and extract <function-id>/<model-name> from the
      request model field.
    • Apply the existing NVCF authorization, model URI allowlist, routing method,
      rate limiting, session affinity, tracing, and metrics behavior.
    • Rewrite only the routed model prefix and proxy the Anthropic request and
      response formats without converting content blocks or SSE events.
    • Preserve protocol headers such as anthropic-version and supported
      anthropic-beta values. Define an explicit forwarded-header policy and
      strip the reported gateway-only extra-headers metadata before the request
      reaches the model backend.
    • Account for Anthropic input_tokens and output_tokens when usage is
      available, and safely release reservations when it is not.
  • LLM request router (Stargate):
    • Add /v1/messages to the HTTP proxy surface.
    • Preserve path/query, body, Anthropic headers, streaming response events,
      upstream status, and error bodies over both direct and reverse tunnels.
    • Continue stripping hop-by-hop and internal routing/retry headers.
  • Vanity Gateway:
    • Add a Messages route and configuration section, including LLM Gateway
      targets, model lookup, custom-header validation, request sizing, telemetry,
      and shadowing behavior where supported.
  • Model and release configuration:
    • Treat /v1/messages as a supported llmConfig.uris value.
    • Publish compatible LLM API Gateway and Stargate images/charts and update the
      self-managed stack catalog so both sides are upgraded together.
    • Document configuration and a public curl example.

Acceptance criteria:

  • A model that declares /v1/messages can serve both streaming and
    non-streaming native Messages requests through the LLM invocation endpoint.
  • Requests for a model that does not declare /v1/messages follow the current
    model URI allowlist mode.
  • Basic text, system prompts, tool use and tool-result follow-up, thinking
    content blocks, token limits, and Anthropic SSE events round-trip without an
    OpenAI protocol conversion.
  • anthropic-version and supported beta headers reach the backend, while
    gateway-only extra-headers metadata and internal NVCF routing headers do
    not leak into the model API.
  • Upstream 4xx/5xx status, Anthropic error bodies, response headers, client
    cancellation, and long-running streams propagate correctly.
  • Unit and integration tests cover LLM API Gateway routing and accounting,
    Stargate direct and reverse tunnels, Vanity Gateway configuration/routing,
    and a Claude Code-compatible tool loop smoke test.
  • Metrics and spans use the stable /v1/messages endpoint label without
    adding model names, request IDs, or other unbounded metric labels.

Describe alternatives you've considered

An external proxy can translate /v1/messages to /v1/chat/completions.
That is useful as a temporary workaround, but it adds another operational hop
and can lose Anthropic-specific behavior used by agent clients. A native
pass-through path provides a smaller and more faithful compatibility surface.

Additional context

No duplicate issue was found by searching this repository for /v1/messages,
Anthropic, Messages API, or Claude Code.


By submitting this issue, you agree to follow our
code of conduct and our
contributing guidelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/invocation-plane-services/llm-api-gateway/api/routes.go, src/libraries/rust/stargate/crates/stargate/src/http_proxy.rs, and src/invocation-plane-services/vanity-gateway/gateway/h2.go, then review docs/user/llm-gateway.md and the related configuration models. Trace existing Chat Completions routing, proxying, and release configuration before identifying corresponding tests. Done means native streaming and non-streaming /v1/messages requests pass through all named layers with headers, errors, accounting, and configuration behavior covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
api, backend, cloud, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.