livepeer / livepeer/livepeer-python-gateway
TricklePublisher.SegmentWriter masks CancelledError as TrickleSegmentWriteError on shutdown
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Symptom
Normal session shutdown produces tracebacks that look like errors but happen on every clean teardown:
livepeer_gateway.trickle_publisher.TrickleSegmentWriteError:
Trickle segment writer timed out after 5.0s
Trickle segment close suppressed seq=0
...
TimeoutError
Root cause (revised after investigation)
Not an asyncio.wait_for cancellation race (initial hypothesis was wrong). Verified via timestamped logs after migrating to asyncio.timeout — the timeouts still fire because they're legitimate:
- Mid-stream, the HTTP post-body consumer for the active segment becomes wedged or closed (the orchestrator-side reader stops reading, or the connection drops).
SegmentWriter's internal queue fills up.- The next
await self.queue.put(data)in_stream_pipe_to_trickleblocks waiting for queue space. - After 5s, the timeout fires →
TrickleSegmentWriteError→MediaPublishlogs"dropped segment ... mid-stream"and enters drain mode. - On
/stream/stop,SegmentWriter.close()callsqueue.put(None)to signal end-of-stream — but the consumer is gone, so it hangs. - After 5s the timeout fires →
"Trickle segment close suppressed"warning logged.
So during shutdown, every active segment with a wedged consumer adds 5s + a misleading-looking traceback before teardown completes. PR #7 / live_grayscale/test.sh reproduces it consistently.
Proposed fix (needs work)
SegmentWriter needs to know when its associated HTTP post-task is dead and short-circuit close() / write() rather than block on queue.put forever. Options:
- Track the post-task on the segment; on
close()checkpost_task.done()beforequeue.put(None). - Use
put_nowait()and absorbQueueFull. - Reduce the close-path timeout to e.g. 0.5s (still misleading but faster).
What PR #11 does not fix
PR #11 migrates wait_for → asyncio.timeout which is modernization, not a fix for this issue. The shutdown noise persists after that migration; investigation here is separate.
Reproduction
cd examples/runner/live_grayscale
docker compose up -d --wait --build
SKIP_VIEWER=1 bash test.sh # passes
docker logs live_grayscale # shows the tracebacks during shutdown
Severity
Cosmetic — teardown completes correctly, but the noise causes false alarms during debugging and adds 5s × N-active-segments to shutdown latency.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SegmentWriter.write(), close(), and _stream_pipe_to_trickle, then reproduce with live_grayscale/test.sh after running the documented Docker Compose setup. Trace the associated HTTP post-task and queue behavior during shutdown; done means clean teardown no longer emits misleading tracebacks or waits 5 seconds per active segment while the reproduction still passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- audio-video-rtc, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100