[Feature]: Engine-agnostic credential so PAIR can talk to an authenticated local engine

Area

Engine or model management

User problem

PAIR has no way to hold a credential for the local inference engine it manages, so enabling the engine's own authentication makes the engine invisible to PAIR and takes the cluster's routing offline.

Local engines increasingly ship a bearer-token auth option: LM Studio has "Require Authentication" with a token manager in its server settings, llama.cpp's server has --api-key, and Ollama deployments are commonly fronted by a reverse proxy that requires one. These are first-class, documented features of the engines PAIR supports.

Today, turning any of them on breaks PAIR completely. Observed on 0.1.1 (13b6811), built from source with make build-services, nvpair-ui-broker 0.40.2:

Request	Result
Engine directly, with credential	200
Engine directly, no credential	401
PAIR proxy /v1/models, no credential	503 {"error":"model inventory unavailable"}
PAIR proxy /v1/models, credential in Authorization	503 {"error":"model inventory unavailable"}
PAIR Ollama proxy /api/tags	503 {"error":"model inventory unavailable"}

nvpair-engine-manager reaches the engine over plain loopback HTTP with no credential — fmt.Sprintf("http://127.0.0.1:%d%s", port, path) in services/nvpair-engine-manager/actions.go:114 and pull.go:98. The engine answers 401, inventory comes back empty, and because the proxies only route to a node whose inventory advertises the exact requested model, nothing is ever eligible.

The blast radius is cluster-wide, not local to the affected node. Any peer that would have routed to that engine also returns 503, since the models it was relying on have disappeared from the merged inventory.

Two things make this hard to self-diagnose:

• Supplying the credential as a client Authorization header on the proxy does not help. The request fails at the eligibility gate before any forwarding, so the header never reaches the engine.
• The error names the wrong subsystem. model inventory unavailable points at inventory, not at the 401 that caused it. services/lmstudio-proxy/proxy.go:1005 documents 401 as a "genuine client error" that is deliberately not retried, which is the correct call for a client mistake but hides an engine-side auth failure.

There is no existing mechanism to work around. A repo-wide code search finds zero hits for apiKey, api_key, or LMSTUDIO_API_KEY. Every Authorization hit in the tree belongs to CORS allow-headers, cluster mTLS/truststore, tests, or a vendored RFC text — none touch the engine hop. The docs, including known-issues.mdx and troubleshooting.mdx, never mention engine authentication. The only available workaround is to turn the engine's auth back off.

Desired outcome

An engine-agnostic credential that PAIR carries on every engine-bound request, rather than anything specific to one engine.

The engine registry is already the right seam and is already engine-agnostic — services/nvpair-engine-manager/registry.go:142 models an engine as a URL template (http://127.0.0.1:{port}/) and probes both OpenAI-shaped (/v1/models, /api/v1/models) and Ollama-shaped (/api/tags, /api/ps) surfaces. A per-engine credential field there would cover every current and future engine at once, instead of adding an LM Studio-specific setting that Ollama and a bare llama.cpp server would each need duplicated later.

Concretely:

• A per-engine credential, configurable by flag and environment variable, and surfaced wherever engines are configured (settings and the TUI's Engines tab).
• Applied on every request PAIR makes to that engine: inventory probes, the control operations in controlmodels.go / controlstream.go (/v1/models/load, /unload, /delete, /pull), and inference forwarding in both lmstudio-proxy and ollama-proxy. A credential that covers inference but not inventory would still leave the node unroutable.
• Sent as Authorization: Bearer <value> by default, since that is what LM Studio, llama.cpp --api-key, and typical Ollama reverse proxies all accept. A configurable header name would future-proof engines that differ.
• A distinguishable error when the engine rejects the credential. An engine returning 401 to PAIR should surface as an engine authentication failure, not as model inventory unavailable.

Both proxies are near-identical clones sharing routing, failover, and CORS code, so this should land once in shared code and apply to both.

Alternatives considered

• Forward the client's Authorization header to the engine. Rejected: it does not fix inventory, which is fetched by nvpair-engine-manager outside any client request, and inventory is what gates routing. It would also mean every client needs the engine's credential, which defeats PAIR being the front door.
• Turn the engine's authentication off. The current de facto workaround. It is not viable for anyone who enabled engine auth deliberately, and it silently narrows what PAIR can sit in front of.
• An LM Studio-specific setting. Narrower to implement, but the same gap already exists for llama.cpp --api-key and proxied Ollama, so it would need duplicating per engine. The registry abstraction makes the generic version roughly the same amount of work.

Compatibility and security implications

• Fully backward compatible. No credential configured means today's behavior exactly — no header added.
• The credential must not cross the cluster boundary. It authenticates a node to its own loopback engine. It should never be forwarded to peers or included in anything advertised over mDNS. Peer-to-peer auth is already handled by cluster mTLS pinning, and services/lmstudio-proxy/ingress.go:87-93 documents the mTLS pin as "the sole authorization boundary," forwarding trusted peers straight to the loopback engine — that model is unchanged, since the receiving node applies its own credential locally.
• Storage alongside existing per-user material in the cluster/data dir, at the same file permissions as other node secrets.
• Redaction: it must be excluded from logs and from the log sanitizer in resources/scripts/ referenced by SUPPORT.md.
• No wire-format or API-compatibility change: this only adds a header on requests PAIR already makes.

Validation approach

1. Start a supported engine with authentication enabled (LM Studio "Require Authentication", or llama.cpp server with --api-key). Confirm the engine returns 401 unauthenticated and 200 with the credential.
2. Without configuring PAIR, confirm the current failure: GET /v1/models on the PAIR proxy returns 503 {"error":"model inventory unavailable"}, on that node and on any peer that would route to it.
3. Configure the credential in PAIR. Expect GET /v1/models on the proxy to list the engine's models again, and POST /v1/chat/completions against one of those model IDs to complete — from the engine-bearing node and from a peer routing to it.
4. Exercise a control operation (model load or unload) to confirm the credential is applied outside the inference path.
5. Configure a deliberately wrong credential. Expect an error that identifies engine authentication as the cause, distinct from an empty-inventory error.
6. Confirm the credential appears in no log output and in no sanitizer bundle, and that a peer node never receives it.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

Bắt đầu bằng cách lần theo các yêu cầu đến engine qua services/nvpair-engine-manager/registry.go, actions.go và pull.go, sau đó tiếp tục với controlmodels.go, controlstream.go, lmstudio-proxy và ollama-proxy. Hoàn tất có nghĩa là các yêu cầu inventory, control và inference đã xác thực hoạt động, thông tin xác thực sai tạo ra một lỗi riêng biệt, thông tin xác thực được giữ cục bộ và được redacted, đồng thời các kịch bản validation được liệt kê đều vượt qua.