huggingface / huggingface/datasets
MetadataConfigs drops parquet shards when exported config rows are non-consecutive
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Bug
`_from_exported_parquet_files_and_dataset_infos()` uses `itertools.groupby()` on unsorted `exported_parquet_files`.
If the same `config` appears again after another config, the earlier shard URLs are lost.
## Repro
```python
from itertools import groupby
from operator import itemgetter
exported = [
{"config": "default", "split": "train", "url": ".../train-0000.parquet"},
{"config": "other", "split": "train", "url": ".../other/train.parquet"},
{"config": "default", "split": "train", "url": ".../train-0001.parquet"},
]
metadata_configs = {
config_name: [p["url"] for _, ps in groupby(rows, itemgetter("split")) for p in ps]
for config_name, rows in groupby(exported, itemgetter("config"))
}
# default keeps only train-0001
```
## Proof (failing before fix)
```
FAILED - default only had 1 URL instead of 2
```
## Fix
Sort by `(config, split)` before both `groupby()` calls.
I'll follow up with a PR + regression test.
Contributor guide
Assessment
This issue has not been assessed yet.