VideoReader.get_batch() hangs forever (100% CPU, single thread) on certain H.264 MP4 files, while vr[i] on the same indices works
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 232
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
On a small fraction of otherwise-ordinary H.264/AAC MP4 files, `VideoReader.get_batch(indices)`
never returns. The process spins at ~100% CPU on one thread indefinitely (observed >52 minutes;
in an earlier production run, **38 hours**). It is not interruptible by `SIGTERM` — only `SIGKILL`
terminates it.
Crucially, **`vr[i]` on the exact same indices returns normally on the same file**, so the file is
decodable; only the batched path hangs.
No CUDA, no torch, no GPU is involved — plain decord on CPU (`ctx=cpu(0)`) reproduces it.
## Minimal reproduction
```python
import numpy as np
from decord import VideoReader, cpu
vr = VideoReader("-M8qt69nljE_000030.mp4", ctx=cpu(0))
n = len(vr) # -> 450, returns fine
idx = np.linspace(0, n - 1, 4, dtype=int) # -> [0, 149, 299, 449]
vr.get_batch(idx) # -> never returns
```
Control — the **same file**, same indices, via `__getitem__`, returns immediately:
```python
vr = VideoReader("-M8qt69nljE_000030.mp4", ctx=cpu(0))
n = len(vr)
vr[0]; vr[n - 1] # -> OK
```
Determinism: **3/3 runs hang** (killed by a 40s timeout each time, never reaching the next
statement). A control video with the same codec parameters returns `(4, 720, 1280, 3)` in ~1s.
## Captured stack (Python-level)
Obtained with `faulthandler.dump_traceback_later(300, repeat=True, exit=False)`:
```
Thread 0x00007f79649d3280 (most recent call first):
File ".../site-packages/decord/_ffi/_ctypes/function.py", line 173 in __call__
File ".../site-packages/decord/video_reader.py", line 175 in get_batch
...
```
i.e. it is blocked inside the C FFI call made by `get_batch`, with no further Python frames.
All other threads are idle (`tqdm` monitor in `wait`, etc.).
## Environment
| component | version |
|---|---|
| decord | **0.6.0** (pip) |
| Python | 3.10.20 |
| OS / kernel | Ubuntu, Linux **5.15.0-139-generic** |
| ffmpeg (system) | 4.2.7 |
| CPU | x86_64, 64 cores |
| GPU | not involved (`ctx=cpu(0)`; also reproduces with no CUDA context at all) |
## Affected-file characteristics
The failing file is **not** unusual — its stream parameters are identical to many files that work:
```
codec_name=h264 profile=Main level=31 pix_fmt=yuv420p
width=1280 height=720 has_b_frames=1 refs=1 is_avc=true nal_length_size=4
r_frame_rate=30000/1001 avg_frame_rate=30000/1001 nb_frames=450
format=mov,mp4,m4a,3gp,3g2,mj2 format_duration=15.02s
audio: aac LC
```
**Observations (offered as hints only — we have NOT established causation):**
- Both files we confirmed hanging have `nb_frames=450` at `~29.97` fps.
- Both show a large gap between the **video stream** duration (`10.04s`) and the **container/format**
duration (`~15.02s`), delta ≈ 5.0s. However, files with a smaller but non-zero gap
(e.g. 1.66s, 3.44s) decode fine, so this is at most a contributing factor, not a clean predictor.
- Container metadata is otherwise self-consistent (`nb_frames` ≈ `format_duration × fps`).
## Incidence
Sweeping a corpus of **9,982** such MP4s (each probed in an isolated subprocess with a 30s timeout,
because the hang cannot be interrupted in-process):
- hangs: ****21**** (****0.210%**, 21 / 9,982**)
- the remainder decode via `get_batch` normally
(Exact list of affected filenames available.)
## Why this is easy to miss
A health check that probes `vr[0]` / `vr[len(vr)-1]` reports the corpus as 100% healthy, because
`__getitem__` succeeds on the very files where `get_batch` hangs. Only a probe that calls
`get_batch` — the API production code actually uses — surfaces the problem.
## Impact
A single affected file wedges the consuming process permanently. In our case a training job sat at
100% CPU / 0% GPU for **38 hours** before being killed, producing no output and no error. Because
the hang is in a C call, it is immune to Python-level timeouts and to `SIGTERM`; only process
isolation with `SIGKILL` bounds it.
## Suggested mitigations for users (already applied on our side)
- Probe with the **same** call production uses (`get_batch`), in an **isolated subprocess** with a
hard timeout, and exclude files that hang.
- Do not rely on `signal.alarm` / thread-based timeouts — the C call does not yield.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with decord/video_reader.py get_batch and follow the C FFI call shown in the captured stack into the native backend. Reproduce using the affected MP4 and indices [0, 149, 299, 449], then compare with vr[i]; done means get_batch returns normally without an infinite CPU-bound hang and the regression is covered by a reproducible test or fixture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100