apache / apache/apisix

feat: As a user, I want to route to The Grid through ai-proxy, so that I can consume market-priced inference from APISIX

Open
#13,919 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

### Description

As a user of `ai-proxy`, I want to route to [The Grid](https://thegrid.ai) as a first-class provider, so that I can buy market-priced inference through APISIX the same way I route to OpenAI or OpenRouter today.

The Grid is an OpenAI-compatible inference API. Its model ids are market instruments — a task type and a quality tier (`text-standard`, `code-prime`, `agent-max`) or a lab-scoped market (`claude-opus-latest`, `kimi-latest`) — rather than fixed model names, and requests are filled by whichever supplier is competitive at the time. `/v1/chat/completions` and `/v1/responses` are both served, so it fits the existing `openai-chat` and `openai-responses` capabilities without a new protocol.

Adding the provider module itself is routine and mirrors `openai.lua`. One thing about The Grid does not fit the transport as it stands, and I would like to check the approach before sending it.

### The routing redirect

The Grid's Consumption API validates the request, then answers `307 Temporary Redirect` with a `Location` pointing at its routing layer **on the same host** (`api.thegrid.ai/v1/...` → `api.thegrid.ai/r/v1/...`), where inference is actually fulfilled. This is documented and intentional, not an error condition: https://thegrid.ai/docs/api-reference/request-routing-and-redirects

Most clients follow it transparently, so this is invisible to normal SDK users. `lua-resty-http` does not, so `ai-proxy` would hand the `307` back downstream. That is worse than it first looks:

- `ai-rate-limiting` meters **zero** tokens for every request, because the gateway never sees a response body containing `usage`.
- `ai-proxy-multi` health checks and instance fallback never trigger, since from the Plugin's point of view every request succeeded.
- The `Location` URL is self-authenticating — I confirmed it returns `200` when replayed with **no** `Authorization` header — so the gateway would effectively be converting its configured provider credentials into a bearer URL handed to the caller.

### Proposal

Add opt-in redirect following to `apisix/plugins/ai-transport/http.lua`:

- A provider declares `follow_redirects = true`; every existing provider is untouched and keeps today's behaviour.
- Exactly one hop, `307`/`308` only. `301`/`302`/`303` are excluded because they permit rewriting the method to `GET`, which would silently drop the prompt.
- Method, body and headers are replayed unchanged; the redirect's own query string replaces the original when present, so `auth.query` providers stay authenticated otherwise.
- **Same-origin only.** A cross-origin hop would replay the provider credentials in `Authorization` against a host named by the response rather than by the configuration, so it is refused and surfaces as a request error.

Following it inside the gateway is what keeps token accounting, retries and fallback working, which is the reason to do this in the transport rather than document it as a caveat.

I have this implemented with tests (same-origin hop followed and the model output returned; cross-origin hop refused; mock endpoints added to `t/lib/server.lua` that assert the replay preserved method, body, target query and headers) and will open the PR referencing this issue. Happy to split the transport change out into its own PR, or to take a different approach to the redirect, if you would prefer.

### Environment

- APISIX version: `master`

Contributor guide

Open the contributing guide

Research direction

Read apisix/plugins/ai-transport/http.lua and compare the provider declaration with openai.lua. Run the existing transport tests, then review the mock endpoints in t/lib/server.lua for same-origin 307/308 replay, cross-origin refusal, preserved method, body, headers, and query. Done means the tested redirect behavior supports The Grid without changing existing providers.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
api, security, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.