livepeer / livepeer/go-livepeer

Gateway crashes with a `panic: runtime error` when its Orchestrator disappears while transcoding a stream

Open
#3,784 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: triage
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

Describe the bug

When processing a stream being transcoded via an Orchestrator, the Gateway crashes with panic: runtime error: invalid memory address or nil pointer dereference when the Orchestrator process becomes unavailable.

To Reproduce

Steps to reproduce the behaviour:

  1. Open 3 terminal windows
  2. In first terminal window run an offchain Gateway with:
./livepeer -gateway -orchAddr 127.0.0.1:8935 -transcodingOptions P144p30fps16x9 -v 99
  1. In second terminal window run an offchain Orchestrator/Transcoder with:
./livepeer -orchestrator -transcoder -serviceAddr 127.0.0.1:8935 -v 99
  1. In third terminal window run an ffmpeg Publisher with:
ffmpeg -re -f lavfi -i testsrc=size=640x360:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 1000k        -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1935/test_source
  1. In the second terminal window, press ctrl-C to cancel the Orchestrator process.
  2. In the first terminal window, observe that the Gateway process has crashed, reporting the following:
E1018 11:16:37.577703   72365 segment_rpc.go:557] orchSessionID=212063a9 orchestrator=https://127.0.0.1:8935 Unable to submit segment orch=https://127.0.0.1:8935 orch=https://127.0.0.1:8935 uploadDur=400.712µs err="Post \"https://127.0.0.1:8935/segment\": dial tcp 127.0.0.1:8935: connect: connection refused"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2e1ed85]

goroutine 719 [running]:
github.com/livepeer/go-livepeer/server.NewSessionManager.func1({0x0?, 0xc000c379a0?})
	/__w/go-livepeer/go-livepeer/server/broadcast.go:540 +0x25
github.com/livepeer/go-livepeer/server.(*SessionPool).removeSession(0xc001576200, 0xc000a64300)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:453 +0x9e
github.com/livepeer/go-livepeer/server.(*BroadcastSessionsManager).suspendAndRemoveOrch(0xc000a2a5c0, 0xc000a64300)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:569 +0xa5
github.com/livepeer/go-livepeer/server.transcodeSegment({0xb3b100, 0x4bc0e20}, 0xc0003b00f0, 0xc000a9e180, {0xc000c900f0, 0xb}, 0x0, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1180 +0x748
github.com/livepeer/go-livepeer/server.processSegment({0xb3b100, 0x4bc0e20}, 0xc0003b00f0, 0xc000a9e180, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1074 +0xc85
created by github.com/livepeer/go-livepeer/server.(*LivepeerServer).StartMediaServer.gotRTMPStreamHandler.func5.1.1 in goroutine 586
	/__w/go-livepeer/go-livepeer/server/mediaserver.go:437 +0xd2

Expected behavior

In the context of an offchain Gateway, at step 6, I expect the Gateway to continue to ingest and egress the stream, albeit without transcoded content. It ought to handle the disappearance of the Orchestrator gracefully.

In the context of an onchain Gateway, it ought to be able to gracefully discover and employ another Orchestrator from the active set, in order to avoid unnecessary discontinuity of the playback for Consumers viewing the transcoded content.

Screenshots

Image

Environment

Livepeer Node Version: 0.8.8
Golang runtime version: gc go1.23.2
Architecture: amd64
Operating system: linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in server/broadcast.go at the stack-trace locations around NewSessionManager, SessionPool.removeSession, suspendAndRemoveOrch, and transcodeSegment. Reproduce the failure by stopping the Orchestrator during transcoding, then trace the unavailable-Orchestrator path. Done means the Gateway no longer panics and handles the stream as described for offchain and onchain cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.