facebookresearch / facebookresearch/jepa

video_dataset: Random sample assignment

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
422
PR merge metrics
No merged PRs in 30d

Description

In the file video_dataset.py, the __getitem__ not make sense to me:

```python
def __getitem__(self, index):
sample = self.samples[index]

# Keep trying to load videos until you find a valid sample
loaded_video = False
while not loaded_video:
buffer, clip_indices = self.loadvideo_decord(sample) # [T H W 3]
loaded_video = len(buffer) > 0
if not loaded_video:
index = np.random.randint(self.__len__())
sample = self.samples[index]

# Label/annotations for video
label = self.labels[index]

def split_into_clips(video):
""" Split video into a list of clips """
fpc = self.frames_per_clip
nc = self.num_clips
return [video[i*fpc:(i+1)*fpc] for i in range(nc)]

# Parse video into frames & apply data augmentations
if self.shared_transform is not None:
buffer = self.shared_transform(buffer)
buffer = split_into_clips(buffer)
if self.transform is not None:
buffer = [self.transform(clip) for clip in buffer]

return buffer, label, clip_indices

```

Particularly, the following:

```python
if not loaded_video:
index = np.random.randint(self.__len__())
sample = self.samples[index]
```

In the current setup (at least in eval), samples are file paths to videos. So, here we´re replacing the video output with a random other video and returning it as the video at the current index with the label for the current index?

Worst case:
This could mess up validation (if the labels are used)
Best case:
Random double videos

Or maybe I´m missing something?

Contributor guide

Open the contributing guide

Research direction

Start in video_dataset.py at __getitem__ and trace loadvideo_decord, self.samples, and self.labels. Check how an invalid video changes the selected index and whether the returned label remains aligned, especially during evaluation. Done means the intended fallback behavior is documented or corrected and its validation impact is demonstrated.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.