facebookresearch / facebookresearch/SlowFast
Bug: target_fps in the decoder shouldn't be an integer in NTSC system.
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
When I was visualising the Kinetics dataloader (but modified a little bit to use EPIC-Kitchens), I noticed that always the first and the second frame from the decoder is the exact same output when you are using sampling rate of 1.
Note that:
1. If sampling rate is 2 or higher, you may not notice this artefact. However, the first two frames would have a different sampling rate (slightly slower speed in movement).
2. If you're using noisy data (like Kinetics) that already has a lot of frame duplication due to frame rate conversion, you may not notice this.
3. EPIC-Kitchens have a very clean source with most videos in 60000/1001 (~59.94) fps. And it's obvious that all the videos from the dataloader have duplicated first two frames.
I changed the target_fps from 30 to 30000/1001 (~29.97), it works without the duplicated image.
In `decoder.py`,
https://github.com/facebookresearch/SlowFast/blob/38dfcb037fd220b63eb73d58b9ef2aec8ed672ac/slowfast/datasets/decoder.py#L239-L246
The target_fps is 30 and fps is 30000/1001 (29.97...) which is causing the problem. Possibly, the fps/target_fps is 29.97/30 which is less than one, and when you're converting that to integer (in get_start_end_idx), it will be zero and it's ignoring one of the frames.
My visualisation code looks like below. `VideoClassificationDataset` is a mere copy of Kinetics.py, with removing the cfg dependencies and using a different CSV format for my own use. I converted the EPIC-Kitchens to 15000/1001 fps using FFmpeg (because I wanted the decoding speed to be faster) and used 32x1 instead of 32x2 in 29.97 fps.
```python
import torch
from PIL import Image
import numpy as np
#from dataloaders.DALI_video_loader import DALILoader
from slowfast.datasets.video_classification_dataset import VideoClassificationDataset
if __name__ == '__main__':
batch_size = 4
input_frame_length = 32
input_frame_stride = 1
input_target_fps = 15000 / 1001 # 14.985014985014985
mean = 0.45
std = 0.225
# Dataset
val_dataset = VideoClassificationDataset("/home/kiyoon/datasets/EPIC_KITCHENS_2018/epic_list/train.csv", "train",
input_frame_length, input_frame_stride, target_fps=input_target_fps, enable_multi_thread_decode=False, decoding_backend='pyav')
val_dataloader = torch.utils.data.DataLoader(val_dataset, batch_size=batch_size, shuffle=False, sampler=None, num_workers=4, pin_memory=False, drop_last=False)
val_dataloader_it = iter(val_dataloader)
#data = next(val_dataloader_it)
#data = next(val_dataloader_it)
#data = next(val_dataloader_it)
#data = next(val_dataloader_it)
data = next(val_dataloader_it)
inputs, uids, labels, _, _ = data
inputs = inputs.numpy()
print(uids)
inputs = inputs.transpose((0,2,3,4,1)) # B, F, H, W, C
batch_array = (inputs * std + mean) * 255
batch_array = batch_array.astype(np.uint8)
print(batch_array.shape)
for frame_num, img_array in enumerate(batch_array[3]):
#img_array = batch_array[0, 0]
img = Image.fromarray(img_array)
img.save('{:02d}.png'.format(frame_num))
```
## In sum,
Possible target_fps values are 15000/1001, 24000/1001, 30000/1001, 60000/1001, 25, 50, etc. Otherwise, you'll see duplicates in frames.
Contributor guide
Assessment
This issue has not been assessed yet.