livepeer / livepeer/lpms

Transcoding hangs under load

Open
#158 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
283
Forks
72
Avg merge
27m
Merged PRs (30d)
1

Description

While doing throughput testing hit situation where transcoding hangs (reproduced three times).
It was transcoding one 10 minutes generated video to seven 720 renditions in ten streams simultaneously. Looking at output files, it completely transcoded most of the files, except some. `nvidia-smi` shows that `Livepeer` process still uses GPU, but output files (one not completed) not growing in size.
There is stopped GCE instance on which I've saw this, I've left it intact.
To reproduce:
1. Start `Ivan-gpu-p100` instance
2. got to `/home/dark/go-livepeer`
3. run `bench.sh`
4. wait 12 minutes
5. look at the `/disk-1-temp` directory - there will be 70 output files, most complete (same size), some smaller. and that smaller files not growing in size.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with /home/dark/go-livepeer/bench.sh on the stopped Ivan-gpu-p100 GCE instance, then inspect /disk-1-temp and nvidia-smi during the reproduced load. Compare the incomplete output files with the completed ones and collect relevant process or transcoding logs. Done means the benchmark completes without stalled streams and all expected output files finish growing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, shell
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.