Apparent locking issues when running across multiple GPUs
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 232
- PR merge metrics
- No merged PRs in 30d
Description
I've noticed an interesting issue when running on multi-GPU machines: although selecting `gpu(N)` as the decoding context initially works as expected, the overall throughput when running multiple processes drops off very rapidly until there's only one process showing activity on a single GPU, sometimes with occasional very short bursts of processing from others.
This happens even when the processes are totally independent (started separately from different `screen` sessions, operating on entirely different files, using separate GPUs, for example), which leads me to think there's probably a hardware- or system-level locking mechanism being used globally rather than per-process since it occurs even between separate python instances.
Working theory is that it could be falling through to a global lock of some kind due to setting `decoder_info_.vidLock = nullptr;`, but so far that hasn't brought us closer to a fix. Would be very helpful to hear if anyone else has (or hasn't!) run into similar issues?
Possibly related to https://github.com/dmlc/decord/issues/187 and/or https://github.com/dmlc/decord/issues/159?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.