dmlc / dmlc/decord

VideoReader.get_batch() hangs forever (100% CPU, single thread) on certain H.264 MP4 files, while vr[i] on the same indices works

Open
#375 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
232
PR merge metrics
No merged PRs in 30d

Description

## Summary

On a small fraction of otherwise-ordinary H.264/AAC MP4 files, `VideoReader.get_batch(indices)`
never returns. The process spins at ~100% CPU on one thread indefinitely (observed >52 minutes;
in an earlier production run, **38 hours**). It is not interruptible by `SIGTERM` — only `SIGKILL`
terminates it.

Crucially, **`vr[i]` on the exact same indices returns normally on the same file**, so the file is
decodable; only the batched path hangs.

No CUDA, no torch, no GPU is involved — plain decord on CPU (`ctx=cpu(0)`) reproduces it.

## Minimal reproduction

```python
import numpy as np
from decord import VideoReader, cpu

vr = VideoReader("-M8qt69nljE_000030.mp4", ctx=cpu(0))
n = len(vr) # -> 450, returns fine
idx = np.linspace(0, n - 1, 4, dtype=int) # -> [0, 149, 299, 449]
vr.get_batch(idx) # -> never returns
```

Control — the **same file**, same indices, via `__getitem__`, returns immediately:

```python
vr = VideoReader("-M8qt69nljE_000030.mp4", ctx=cpu(0))
n = len(vr)
vr[0]; vr[n - 1] # -> OK
```

Determinism: **3/3 runs hang** (killed by a 40s timeout each time, never reaching the next
statement). A control video with the same codec parameters returns `(4, 720, 1280, 3)` in ~1s.

## Captured stack (Python-level)

Obtained with `faulthandler.dump_traceback_later(300, repeat=True, exit=False)`:

```
Thread 0x00007f79649d3280 (most recent call first):
File ".../site-packages/decord/_ffi/_ctypes/function.py", line 173 in __call__
File ".../site-packages/decord/video_reader.py", line 175 in get_batch
...
```

i.e. it is blocked inside the C FFI call made by `get_batch`, with no further Python frames.
All other threads are idle (`tqdm` monitor in `wait`, etc.).

## Environment

| component | version |
|---|---|
| decord | **0.6.0** (pip) |
| Python | 3.10.20 |
| OS / kernel | Ubuntu, Linux **5.15.0-139-generic** |
| ffmpeg (system) | 4.2.7 |
| CPU | x86_64, 64 cores |
| GPU | not involved (`ctx=cpu(0)`; also reproduces with no CUDA context at all) |

## Affected-file characteristics

The failing file is **not** unusual — its stream parameters are identical to many files that work:

```
codec_name=h264 profile=Main level=31 pix_fmt=yuv420p
width=1280 height=720 has_b_frames=1 refs=1 is_avc=true nal_length_size=4
r_frame_rate=30000/1001 avg_frame_rate=30000/1001 nb_frames=450
format=mov,mp4,m4a,3gp,3g2,mj2 format_duration=15.02s
audio: aac LC
```

**Observations (offered as hints only — we have NOT established causation):**
- Both files we confirmed hanging have `nb_frames=450` at `~29.97` fps.
- Both show a large gap between the **video stream** duration (`10.04s`) and the **container/format**
duration (`~15.02s`), delta ≈ 5.0s. However, files with a smaller but non-zero gap
(e.g. 1.66s, 3.44s) decode fine, so this is at most a contributing factor, not a clean predictor.
- Container metadata is otherwise self-consistent (`nb_frames` ≈ `format_duration × fps`).

## Incidence

Sweeping a corpus of **9,982** such MP4s (each probed in an isolated subprocess with a 30s timeout,
because the hang cannot be interrupted in-process):

- hangs: ****21**** (****0.210%**, 21 / 9,982**)
- the remainder decode via `get_batch` normally

(Exact list of affected filenames available.)

## Why this is easy to miss

A health check that probes `vr[0]` / `vr[len(vr)-1]` reports the corpus as 100% healthy, because
`__getitem__` succeeds on the very files where `get_batch` hangs. Only a probe that calls
`get_batch` — the API production code actually uses — surfaces the problem.

## Impact

A single affected file wedges the consuming process permanently. In our case a training job sat at
100% CPU / 0% GPU for **38 hours** before being killed, producing no output and no error. Because
the hang is in a C call, it is immune to Python-level timeouts and to `SIGTERM`; only process
isolation with `SIGKILL` bounds it.

## Suggested mitigations for users (already applied on our side)

- Probe with the **same** call production uses (`get_batch`), in an **isolated subprocess** with a
hard timeout, and exclude files that hang.
- Do not rely on `signal.alarm` / thread-based timeouts — the C call does not yield.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with decord/video_reader.py get_batch and follow the C FFI call shown in the captured stack into the native backend. Reproduce using the affected MP4 and indices [0, 149, 299, 449], then compare with vr[i]; done means get_batch returns normally without an infinite CPU-bound hang and the regression is covered by a reproducible test or fixture.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.