huggingface / huggingface/datasets

`DataFilesList.filter()` leaves `origin_metadata` misaligned with the filtered files

Open Beginner friendly
#8,618 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Description

`DataFilesList` keeps `origin_metadata`, a list parallel to the file list where
each entry describes the origin of the file at the same index (e.g. the data
files pattern it was resolved from). After `filter()` removes some files, the
returned `DataFilesList` keeps the *unfiltered* `origin_metadata`, so metadata
no longer lines up with the files it describes: `len(origin_metadata)` is larger
than the number of files, and `origin_metadata[i]` refers to a different file.

## Reproduction

```python
from datasets.data_files import DataFilesList

# two resolved files, each with origin metadata
data_files = DataFilesList.from_patterns(["*"], "/path/to/dir/with/data.txt/and/data.csv")

filtered = data_files.filter(extensions=[".txt"])
assert len(filtered.origin_metadata) == len(filtered) # fails: 2 != 1
```

Any consumer iterating `zip(filtered, filtered.origin_metadata)` or indexing
`filtered.origin_metadata[i]` associates the wrong metadata with a file, and
indexing past `len(filtered)` reads stale metadata.

## Expected behavior

`filter()` should apply the same filtering to `origin_metadata` so the
one-to-one alignment between files and their metadata is preserved.

## Environment

- `datasets` version: 5.0.2.dev0 (main, d336dcb)
- Python: 3.11

Contributor guide

Open the contributing guide

Research direction

Start in datasets/data_files.py, focusing on DataFilesList.filter() and how origin_metadata is stored alongside the file list. Run the provided reproduction with the .txt extension filter, then verify that filtered files and origin_metadata remain the same length and correspond by index.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.