huggingface / huggingface/datasets
`DataFilesList.filter()` leaves `origin_metadata` misaligned with the filtered files
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Description
`DataFilesList` keeps `origin_metadata`, a list parallel to the file list where
each entry describes the origin of the file at the same index (e.g. the data
files pattern it was resolved from). After `filter()` removes some files, the
returned `DataFilesList` keeps the *unfiltered* `origin_metadata`, so metadata
no longer lines up with the files it describes: `len(origin_metadata)` is larger
than the number of files, and `origin_metadata[i]` refers to a different file.
## Reproduction
```python
from datasets.data_files import DataFilesList
# two resolved files, each with origin metadata
data_files = DataFilesList.from_patterns(["*"], "/path/to/dir/with/data.txt/and/data.csv")
filtered = data_files.filter(extensions=[".txt"])
assert len(filtered.origin_metadata) == len(filtered) # fails: 2 != 1
```
Any consumer iterating `zip(filtered, filtered.origin_metadata)` or indexing
`filtered.origin_metadata[i]` associates the wrong metadata with a file, and
indexing past `len(filtered)` reads stale metadata.
## Expected behavior
`filter()` should apply the same filtering to `origin_metadata` so the
one-to-one alignment between files and their metadata is preserved.
## Environment
- `datasets` version: 5.0.2.dev0 (main, d336dcb)
- Python: 3.11
Contributor guide
Research direction
Start in datasets/data_files.py, focusing on DataFilesList.filter() and how origin_metadata is stored alongside the file list. Run the provided reproduction with the .txt extension filter, then verify that filtered files and origin_metadata remain the same length and correspond by index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100