livepeer / livepeer/go-livepeer

Gateway nil-pointer panic when AI request has all orchestrators price-filtered

Open
#3,917 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: triage
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

Summary

When the only orchestrators capable of serving an AI capability are filtered out by the per-capability max price (with --ignoreMaxPriceIfNeeded=false), the AI request handler panics with a nil-pointer dereference at server/ai_process.go:1567 instead of returning an error to the client. The HTTP client sees curl: (52) Empty reply from server.

Repro

livepeer 0.8.9 running as gateway with --aiServiceRegistry and without --ignoreMaxPriceIfNeeded (i.e. default false). Orchestrator pool already populated.

  1. Pick an AI capability whose only available orchestrator charges price P. (In our session: audio-to-text / openai/whisper-large-v3, served by one reachable orch at 13,915,084 wei/pixel-second.)

  2. Set the runtime max price below P:

curl -X POST http://127.0.0.1:5935/setMaxPriceForCapability \
  -d "maxPricePerUnit=10000000" -d "pixelsPerUnit=1" -d "currency=wei" \
  -d "pipeline=audio-to-text" -d "modelID=openai/whisper-large-v3"
  1. Send a request:
curl -X POST http://127.0.0.1:9935/audio-to-text \
  -F "audio=@speech.mp3;type=audio/mpeg" \
  -F "model_id=openai/whisper-large-v3"

Expected

A graceful HTTP error response, e.g. 503 with a "no eligible orchestrators" body.

Actual

curl: (52) Empty reply from server. The gateway accepts the connection, reads the request body, then closes the connection without sending an HTTP response.

Gateway journal shows the price filter rejecting the orch correctly, then the AI session manager handing the same orch to a downstream path that nil-panics:

selection_algorithm.go:86 Orchestrator 0xd18a02647d99dC9F79AfbE0f58f8353178e6141F is above max price 10000000.000, price=13913435.880
selection.go:156          Selected orchestrator  from available list: []
selection.go:156          Selected orchestrator  from available list: [https://lp-orch.j1v.co:8936]
2026/05/01 22:46:18 http: panic serving 127.0.0.1:34340: runtime error: invalid memory address or nil pointer dereference
goroutine 3538 [running]:
panic.go:783 +0x132
go-livepeer/server/ai_process.go:1567 +0x1fb0
go-livepeer/server/ai_process.go:938 +0xa5
go-livepeer/server/ai_mediaserver.go:166 +0x22d

The two consecutive Selected orchestrator from available list: ... log lines are the smoking gun. The first call returns an empty list (price filter applied correctly), the second call returns the just-rejected orch from a fallback list that doesn't apply the price filter. The code at ai_process.go:1567 is not initialized for the case where the fallback orch is unusable, and dereferences a nil pointer.

Suggested fix

Either:

  1. Semantic fix — the AI session manager's fallback list at the second Selected orchestrator site should respect the same filters that gated the primary list (price filter, capability filter, blocklist). If the price filter rejected an orch, the fallback shouldn't reintroduce them.
  2. Defensive guardai_process.go:1567 should detect the nil case and return errNoOrchs (or equivalent) instead of dereferencing.

Option 1 is the correct fix; option 2 is a backstop so one misconfigured request can't crash the goroutine serving it.

Why this matters

--ignoreMaxPriceIfNeeded=true (which many ops set for service continuity) papers over the bug entirely. But it's the wrong default for any operator who wants predictable spend — those operators set strict price caps and will hit this panic the moment market prices move above their cap. The same code path likely fires when --orchBlocklist filters every otherwise-eligible orch as well.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with server/ai_process.go:1567 and trace the two orchestrator-selection calls through selection.go and selection_algorithm.go, comparing how price filtering affects each list. Reproduce the request with --ignoreMaxPriceIfNeeded=false and verify that a fully filtered pool returns an HTTP error such as 503 without reintroducing a rejected orchestrator or panicking.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.