facebookresearch / facebookresearch/seamless_interaction

SMPL-H frame counts in V01_S1602 do not match the documented 30 Hz rate

Open
#34 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
413
Forks
31
PR merge metrics
No merged PRs in 30d

Description

### Have you read the Contributing Guidelines?

- [x] I have read the [Contributing Guidelines](https://github.com/facebookresearch/seamless_interaction/blob/main/CONTRIBUTING.md).

### Issue Type

Format inconsistencies

### Dataset Label

naturalistic

### Dataset Split

train

### Affected Files/Data and Issue Description

While aligning the SMPL-H features with audio, we found six records in `naturalistic/train/0042` whose frame counts do not match the documented 30 Hz rate.

| Record | Archive | SMPL-H frames | WAV duration | Implied rate |
|---|---|---:|---:|---:|
| V01_S1602_I00000125_P1071 | `naturalistic/train/0042/0028.tar` | 80,640 | 336 s | 240 Hz |
| V01_S1602_I00000125_P2560 | `naturalistic/train/0042/0022.tar` | 80,640 | 336 s | 240 Hz |
| V01_S1602_I00000126_P1071 | `naturalistic/train/0042/0053.tar` | 72,000 | 300 s | 240 Hz |
| V01_S1602_I00000126_P2560 | `naturalistic/train/0042/0042.tar` | 57,370 | 300 s | 191.233 Hz |
| V01_S1602_I00000307_P1071 | `naturalistic/train/0042/0005.tar` | 53,760 | 224 s | 240 Hz |
| V01_S1602_I00000307_P2560 | `naturalistic/train/0042/0044.tar` | 53,760 | 224 s | 240 Hz |

We used the length of `smplh:is_valid` as the SMPL-H frame count. The WAV and MP4 durations agree, so the mismatch appears to be specific to the SMPL-H timeline.

The unusual case is `I00000126`: the two participants have different implied rates for the same 300-second interaction. A single interaction-level downsampling factor would therefore not align both records.

### Steps to Reproduce

1. Download the six listed records from `naturalistic/train/0042`.
2. Extract each record's `.npz` and `.wav` files.
3. From the directory containing the extracted files, run:

```python
import wave
import numpy as np

file_ids = [
"V01_S1602_I00000125_P1071",
"V01_S1602_I00000125_P2560",
"V01_S1602_I00000126_P1071",
"V01_S1602_I00000126_P2560",
"V01_S1602_I00000307_P1071",
"V01_S1602_I00000307_P2560",
]

for file_id in file_ids:
with np.load(f"{file_id}.npz", allow_pickle=False) as data:
smplh_frames = len(data["smplh:is_valid"])

with wave.open(f"{file_id}.wav", "rb") as wav:
duration = wav.getnframes() / wav.getframerate()

print(file_id, smplh_frames, duration, smplh_frames / duration)
```

4. The resulting implied rates are 240 Hz for five records and approximately 191.233 Hz for `V01_S1602_I00000126_P2560`, rather than the documented 30 Hz.

### Additional Context

We checked 10,904 records from the Naturalistic Train subset we are using. The other 10,898 records were consistent with approximately 30 Hz. This was not a census of the entire release, but the issue appears highly localized within our subset: all six exceptions come from batch `0042`, session `V01_S1602`, and three interactions.

The records came from six separate official Hugging Face archives. Their SHA-256 hashes match the corresponding Hugging Face LFS hashes, and we inspected the NPZ files before applying any windowing or resampling.

Could you clarify whether these higher and variable rates are intentional? If so, is there per-file timing metadata or a recommended alignment method? If not, will corrected files be released, and are any other sessions known to be affected?

Our current workaround is to exclude these three interactions.

### Self-service

- [x] I'd be willing to help investigate this data issue. Add comment

Contributor guide

Open the contributing guide

Research direction

Start by downloading the six archives listed for naturalistic/train/0042 and rerun the provided Python check against each .npz and .wav pair. Compare the smplh:is_valid lengths with the WAV and MP4 durations, then investigate whether per-file timing metadata explains the differing rates. Done means documenting the cause and recommended alignment method, or identifying the affected files for correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.