livepeer / livepeer/go-livepeer

Orchestrator / Transcoder RPC Timeouts

Open
#576 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: orchestrator QoL status: icebox type: tech debt
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

**Is your feature request related to a problem? Please describe.**

Currently, orchestrators listen to the same port for gRPC, direct bulk uploads, and LPMS downloads. This has worked well so far with broadcasters. However, this runs into a problem when we attempt to expand the orchestrator's interaction to include transcoders on the same port.

There is a read/write timeout of 8 seconds for HTTP requests. While 8 seconds is a reasonable timeout for broadcasters that are expected to be continuously transmitting and receiving requests, transcoders may be connected to an orchestrator for an extended period of time without being assigned work. This does not work well with gRPC streaming, as the connection would time out after 8 seconds of idleness.

**Describe the solution you'd like**

There are a few options, in order of preference:

1. RPC-level keepalive (ping-pong) every few seconds between O/T.
* Generally a useful thing, but pings every ~8 seconds are a bit too frequent
2. Listen to a separate port
* Some benefits with separation of concerns, but more "moving parts" wrt network exposure
3. Close and re-establish the connection every 8 seconds
* Increased traffic (eg, HTTPS handshaking), dropping and re-adding transcoders is slower, harder to establish a baseline for connection reliability
4. Remove the timeout entirely
* Timeouts are a good thing, especially with uncontrolled clients (eg, broadcasters)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the orchestrator/transcoder gRPC connection and the HTTP read/write timeout currently set to 8 seconds. Compare the listed keepalive, separate-port, reconnect, and no-timeout options, then confirm the chosen design preserves appropriate timeout behavior for broadcaster clients.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.