livepeer / livepeer/go-livepeer
Gateway nil-pointer panic when AI request has all orchestrators price-filtered
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 586
- Forks
- 226
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 19
Description
Summary
When the only orchestrators capable of serving an AI capability are filtered out by the per-capability max price (with --ignoreMaxPriceIfNeeded=false), the AI request handler panics with a nil-pointer dereference at server/ai_process.go:1567 instead of returning an error to the client. The HTTP client sees curl: (52) Empty reply from server.
Repro
livepeer 0.8.9 running as gateway with --aiServiceRegistry and without --ignoreMaxPriceIfNeeded (i.e. default false). Orchestrator pool already populated.
-
Pick an AI capability whose only available orchestrator charges price
P. (In our session:audio-to-text/openai/whisper-large-v3, served by one reachable orch at13,915,084 wei/pixel-second.) -
Set the runtime max price below
P:
curl -X POST http://127.0.0.1:5935/setMaxPriceForCapability \
-d "maxPricePerUnit=10000000" -d "pixelsPerUnit=1" -d "currency=wei" \
-d "pipeline=audio-to-text" -d "modelID=openai/whisper-large-v3"
- Send a request:
curl -X POST http://127.0.0.1:9935/audio-to-text \
-F "audio=@speech.mp3;type=audio/mpeg" \
-F "model_id=openai/whisper-large-v3"
Expected
A graceful HTTP error response, e.g. 503 with a "no eligible orchestrators" body.
Actual
curl: (52) Empty reply from server. The gateway accepts the connection, reads the request body, then closes the connection without sending an HTTP response.
Gateway journal shows the price filter rejecting the orch correctly, then the AI session manager handing the same orch to a downstream path that nil-panics:
selection_algorithm.go:86 Orchestrator 0xd18a02647d99dC9F79AfbE0f58f8353178e6141F is above max price 10000000.000, price=13913435.880
selection.go:156 Selected orchestrator from available list: []
selection.go:156 Selected orchestrator from available list: [https://lp-orch.j1v.co:8936]
2026/05/01 22:46:18 http: panic serving 127.0.0.1:34340: runtime error: invalid memory address or nil pointer dereference
goroutine 3538 [running]:
panic.go:783 +0x132
go-livepeer/server/ai_process.go:1567 +0x1fb0
go-livepeer/server/ai_process.go:938 +0xa5
go-livepeer/server/ai_mediaserver.go:166 +0x22d
The two consecutive Selected orchestrator from available list: ... log lines are the smoking gun. The first call returns an empty list (price filter applied correctly), the second call returns the just-rejected orch from a fallback list that doesn't apply the price filter. The code at ai_process.go:1567 is not initialized for the case where the fallback orch is unusable, and dereferences a nil pointer.
Suggested fix
Either:
- Semantic fix — the AI session manager's fallback list at the second
Selected orchestratorsite should respect the same filters that gated the primary list (price filter, capability filter, blocklist). If the price filter rejected an orch, the fallback shouldn't reintroduce them. - Defensive guard —
ai_process.go:1567should detect the nil case and returnerrNoOrchs(or equivalent) instead of dereferencing.
Option 1 is the correct fix; option 2 is a backstop so one misconfigured request can't crash the goroutine serving it.
Why this matters
--ignoreMaxPriceIfNeeded=true (which many ops set for service continuity) papers over the bug entirely. But it's the wrong default for any operator who wants predictable spend — those operators set strict price caps and will hit this panic the moment market prices move above their cap. The same code path likely fires when --orchBlocklist filters every otherwise-eligible orch as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with server/ai_process.go:1567 and trace the two orchestrator-selection calls through selection.go and selection_algorithm.go, comparing how price filtering affects each list. Reproduce the request with --ignoreMaxPriceIfNeeded=false and verify that a fully filtered pool returns an HTTP error such as 503 without reintroducing a rejected orchestrator or panicking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100