facebookresearch / facebookresearch/tribev2

plot_stimuli crashes on silent / speechless video clips

Open Beginner friendly
#46 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
3.2k
Forks
695
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug
`PlotBrain.plot_timesteps(..., show_stimuli=True)` crashes when the input
video clip has no speech or no audio track, because
`BasePlotBrain.plot_stimuli` assumes both (a) `get_audio(...)` returns a
non-None object and (b) every `segment.events` dataframe contains a
`type` column (i.e. at least one `Word` event).

Both assumptions break in realistic workflows — e.g. screen recordings,
silent clips, or any video where WhisperX produces zero words. The model
itself handles the missing-modality case correctly (`main.py` auto-drops
the `text` extractor when no `Word` events exist), so this is strictly
a plotting-side bug.

### Reproduction

```python
import pandas as pd
from pathlib import Path
from tribev2.demo_utils import TribeModel, get_audio_and_text_events

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = get_audio_and_text_events(
pd.DataFrame([{
"type": "Video", "filepath": "silent_clip.mp4",
"start": 0, "timeline": "default", "subject": "default",
}]),
audio_only=True,
)
preds, segments = model.predict(events=df)
model._plotter.plot_timesteps(preds[:5], segments=segments[:5], show_stimuli=True)
# AttributeError: 'NoneType' object has no attribute 'to_soundarray'
# OR KeyError: 'type' when events is None / missing the column
```

### Expected behaviour
plot_stimuli should gracefully skip the audio waveform / word-overlay
subplots when those modalities are not present in the segments, mirroring
the model's existing "auto-drop missing extractor" behaviour.

### Environment
OS: Ubuntu 24.04 aarch64 (NVIDIA DGX Spark, GB10)
Python 3.12.3
torch 2.11.0+cu130 (Blackwell sm_121)

### Proposed fix
Two defensive guards in: `tribev2/plotting/base.py::BasePlotBrain.plot_stimuli`

Only plot the audio waveform when get_audio(...) returns non-None.
Only iterate words when segment.events is not None and the dataframe has a type column.
No change in behaviour when audio / words ARE present.

Contributor guide

Open the contributing guide

Research direction

Start in tribev2/plotting/base.py at BasePlotBrain.plot_stimuli and reproduce the failure with a silent clip or segments lacking word events. Check the audio result and segment.events before plotting those modalities. Done means plot_timesteps(..., show_stimuli=True) skips absent audio and word overlays while preserving behavior when both are present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.