NVIDIA / NVIDIA/Personal-AI-Router

[Bug]: Action HTTP response-header timeout is selected by a hardcoded engine name, so only Ollama gets the long client

Aperta
#25 1 commento 1 reazione 1 assegnatario Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Go
Stelle
1.4k
Fork
250
Merge medio
23h 27m
PR unite (30g)
1

Descrizione

PAIR version or commit

0.91.7 (nvpair-engine-manager 0.17.4, linux/arm64). The code paths below are also present on main.

Affected component

Go services — nvpair-engine-manager

Environment

NVIDIA DGX Spark (GB10, aarch64), Ubuntu 24.04.4 LTS, driver 580.173.02, 10-node PAIR cluster. Reproduced against stock binaries with no local modifications.

Steps to reproduce
  1. Configure an engine other than Ollama.
  2. Issue an engine:action whose HTTP call needs more than 30 seconds to return response headers. The realistic case is a chat action against a large model with a long prompt, where prefill alone can exceed 30s before the first header is written.
  3. Observe the action fail with a transport timeout.
Expected behavior

The per-action timeout should be a property of the action or the manifest rather than of the engine's name, so an engine author can declare that a given action may be slow.

Actual behavior

executor.go defines two clients:

engineResponseHeaderTimeout     = 30 * time.Second
ollamaLoadResponseHeaderTimeout = 10 * time.Minute

and actions.go chooses between them with a literal engine-name comparison:

client := e.client
if engine == "ollama" && action == "run_model" && e.ollamaLoadClient != nil {
    client = e.ollamaLoadClient
}

Every other engine therefore gets 30 seconds for every action, and no manifest field can change it.

By code inspection this is not limited to third-party engines. The shipped lmstudio.json declares a chat action (POST /v1/chat/completions) that uses the 30s client, and the vLLM manifest proposed in #9 declares a chat action as well, so a first-class vLLM engine would inherit the same ceiling.

The failure also presents badly: net/http: timeout awaiting response headers reads as a broken or hung engine rather than a client-side limit.

Sanitized logs or screenshots
action "run_model": Post "http://127.0.0.1:8800/v1/chat/completions":
net/http: timeout awaiting response headers      [30s]

Scope of what I actually observed, to be precise: I hit this with an engine registered through a user manifest in <appdir>/engines/, driving engine-manager over its stdio JSON-RPC. It reproduced consistently while the model was cold and stopped once I pre-warmed the model during load, which is what isolated it to the response-header timeout rather than to the engine itself. I have not reproduced a 30s chat failure on LM Studio directly; that exposure is read from its manifest and the selection logic above.

Suggested fix

An optional per-action timeout_s in the manifest, defaulting to the current 30s, would cover this and would let the Ollama exception move into ollama.json instead of living in Go — removing the engine-name comparison entirely.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.