facebookresearch / facebookresearch/tribev2
plot_stimuli crashes on silent / speechless video clips
- Dominant language
- Jupyter Notebook
- Stars
- 3.2k
- Forks
- 695
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
`PlotBrain.plot_timesteps(..., show_stimuli=True)` crashes when the input
video clip has no speech or no audio track, because
`BasePlotBrain.plot_stimuli` assumes both (a) `get_audio(...)` returns a
non-None object and (b) every `segment.events` dataframe contains a
`type` column (i.e. at least one `Word` event).
Both assumptions break in realistic workflows — e.g. screen recordings,
silent clips, or any video where WhisperX produces zero words. The model
itself handles the missing-modality case correctly (`main.py` auto-drops
the `text` extractor when no `Word` events exist), so this is strictly
a plotting-side bug.
### Reproduction
```python
import pandas as pd
from pathlib import Path
from tribev2.demo_utils import TribeModel, get_audio_and_text_events
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = get_audio_and_text_events(
pd.DataFrame([{
"type": "Video", "filepath": "silent_clip.mp4",
"start": 0, "timeline": "default", "subject": "default",
}]),
audio_only=True,
)
preds, segments = model.predict(events=df)
model._plotter.plot_timesteps(preds[:5], segments=segments[:5], show_stimuli=True)
# AttributeError: 'NoneType' object has no attribute 'to_soundarray'
# OR KeyError: 'type' when events is None / missing the column
```
### Expected behaviour
plot_stimuli should gracefully skip the audio waveform / word-overlay
subplots when those modalities are not present in the segments, mirroring
the model's existing "auto-drop missing extractor" behaviour.
### Environment
OS: Ubuntu 24.04 aarch64 (NVIDIA DGX Spark, GB10)
Python 3.12.3
torch 2.11.0+cu130 (Blackwell sm_121)
### Proposed fix
Two defensive guards in: `tribev2/plotting/base.py::BasePlotBrain.plot_stimuli`
Only plot the audio waveform when get_audio(...) returns non-None.
Only iterate words when segment.events is not None and the dataframe has a type column.
No change in behaviour when audio / words ARE present.
Contributor guide
Research direction
Start in tribev2/plotting/base.py at BasePlotBrain.plot_stimuli and reproduce the failure with a silent clip or segments lacking word events. Check the audio result and segment.events before plotting those modalities. Done means plot_timesteps(..., show_stimuli=True) skips absent audio and word overlays while preserving behavior when both are present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100