livepeer / livepeer/go-livepeer

Malicious Stream Causing GPU Crashes and Node Disruptions

Open
#2,970 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: triage
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

Several orchestrators have noticed, in recent weeks, the existence of one or more streams causing GPU crashes (cuda error) which leads to the loss of all running streams, and even to the crash of the transcoder/node. The stream then switches to another orchestrator, causing the same results.

I am concerned about the impact of this kind of event on the quality of the network, and the possibility that such streams could also be voluntarily injected into the network.

Following the discussion yesterday evening on the watercooler chat, Doug considered the possibility that the Broadcaster node could identify the stream and delete it automatically to prevent it from spreading, but if the stream was injected voluntarily, it might be better to be able to detect it and block it at the orchestrator level.

I have not been able to identify the responsible stream, and this event remains rather rare, but when it happens, several orchestrators usually report it on Discord.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the orchestrator reports and the related Discord discussion to identify the offending stream and determine whether the failure is in the Broadcaster node or orchestrator handling. The issue names no files or tests; the work is complete only when a confirmed detection and prevention approach is defined and validated against the GPU crash and node disruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.