Transcoding hangs under load
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 283
- Forks
- 72
- Avg merge
- 27m
- Merged PRs (30d)
- 1
Description
While doing throughput testing hit situation where transcoding hangs (reproduced three times).
It was transcoding one 10 minutes generated video to seven 720 renditions in ten streams simultaneously. Looking at output files, it completely transcoded most of the files, except some. `nvidia-smi` shows that `Livepeer` process still uses GPU, but output files (one not completed) not growing in size.
There is stopped GCE instance on which I've saw this, I've left it intact.
To reproduce:
1. Start `Ivan-gpu-p100` instance
2. got to `/home/dark/go-livepeer`
3. run `bench.sh`
4. wait 12 minutes
5. look at the `/disk-1-temp` directory - there will be 70 output files, most complete (same size), some smaller. and that smaller files not growing in size.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with /home/dark/go-livepeer/bench.sh on the stopped Ivan-gpu-p100 GCE instance, then inspect /disk-1-temp and nvidia-smi during the reproduced load. Compare the incomplete output files with the completed ones and collect relevant process or transcoding logs. Done means the benchmark completes without stalled streams and all expected output files finish growing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100