facebookresearch / facebookresearch/seamless_interaction
SMPL-H frame counts in V01_S1602 do not match the documented 30 Hz rate
- Dominant language
- Python
- Stars
- 413
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
### Have you read the Contributing Guidelines?
- [x] I have read the [Contributing Guidelines](https://github.com/facebookresearch/seamless_interaction/blob/main/CONTRIBUTING.md).
### Issue Type
Format inconsistencies
### Dataset Label
naturalistic
### Dataset Split
train
### Affected Files/Data and Issue Description
While aligning the SMPL-H features with audio, we found six records in `naturalistic/train/0042` whose frame counts do not match the documented 30 Hz rate.
| Record | Archive | SMPL-H frames | WAV duration | Implied rate |
|---|---|---:|---:|---:|
| V01_S1602_I00000125_P1071 | `naturalistic/train/0042/0028.tar` | 80,640 | 336 s | 240 Hz |
| V01_S1602_I00000125_P2560 | `naturalistic/train/0042/0022.tar` | 80,640 | 336 s | 240 Hz |
| V01_S1602_I00000126_P1071 | `naturalistic/train/0042/0053.tar` | 72,000 | 300 s | 240 Hz |
| V01_S1602_I00000126_P2560 | `naturalistic/train/0042/0042.tar` | 57,370 | 300 s | 191.233 Hz |
| V01_S1602_I00000307_P1071 | `naturalistic/train/0042/0005.tar` | 53,760 | 224 s | 240 Hz |
| V01_S1602_I00000307_P2560 | `naturalistic/train/0042/0044.tar` | 53,760 | 224 s | 240 Hz |
We used the length of `smplh:is_valid` as the SMPL-H frame count. The WAV and MP4 durations agree, so the mismatch appears to be specific to the SMPL-H timeline.
The unusual case is `I00000126`: the two participants have different implied rates for the same 300-second interaction. A single interaction-level downsampling factor would therefore not align both records.
### Steps to Reproduce
1. Download the six listed records from `naturalistic/train/0042`.
2. Extract each record's `.npz` and `.wav` files.
3. From the directory containing the extracted files, run:
```python
import wave
import numpy as np
file_ids = [
"V01_S1602_I00000125_P1071",
"V01_S1602_I00000125_P2560",
"V01_S1602_I00000126_P1071",
"V01_S1602_I00000126_P2560",
"V01_S1602_I00000307_P1071",
"V01_S1602_I00000307_P2560",
]
for file_id in file_ids:
with np.load(f"{file_id}.npz", allow_pickle=False) as data:
smplh_frames = len(data["smplh:is_valid"])
with wave.open(f"{file_id}.wav", "rb") as wav:
duration = wav.getnframes() / wav.getframerate()
print(file_id, smplh_frames, duration, smplh_frames / duration)
```
4. The resulting implied rates are 240 Hz for five records and approximately 191.233 Hz for `V01_S1602_I00000126_P2560`, rather than the documented 30 Hz.
### Additional Context
We checked 10,904 records from the Naturalistic Train subset we are using. The other 10,898 records were consistent with approximately 30 Hz. This was not a census of the entire release, but the issue appears highly localized within our subset: all six exceptions come from batch `0042`, session `V01_S1602`, and three interactions.
The records came from six separate official Hugging Face archives. Their SHA-256 hashes match the corresponding Hugging Face LFS hashes, and we inspected the NPZ files before applying any windowing or resampling.
Could you clarify whether these higher and variable rates are intentional? If so, is there per-file timing metadata or a recommended alignment method? If not, will corrected files be released, and are any other sessions known to be affected?
Our current workaround is to exclude these three interactions.
### Self-service
- [x] I'd be willing to help investigate this data issue. Add comment
Contributor guide
Research direction
Start by downloading the six archives listed for naturalistic/train/0042 and rerun the provided Python check against each .npz and .wav pair. Compare the smplh:is_valid lengths with the WAV and MP4 durations, then investigate whether per-file timing metadata explains the differing rates. Done means documenting the cause and recommended alignment method, or identifying the affected files for correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100