huggingface / huggingface/datasets

MetadataConfigs drops parquet shards when exported config rows are non-consecutive

Open
#8,269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Bug

`_from_exported_parquet_files_and_dataset_infos()` uses `itertools.groupby()` on unsorted `exported_parquet_files`.

If the same `config` appears again after another config, the earlier shard URLs are lost.

## Repro

```python
from itertools import groupby
from operator import itemgetter

exported = [
{"config": "default", "split": "train", "url": ".../train-0000.parquet"},
{"config": "other", "split": "train", "url": ".../other/train.parquet"},
{"config": "default", "split": "train", "url": ".../train-0001.parquet"},
]

metadata_configs = {
config_name: [p["url"] for _, ps in groupby(rows, itemgetter("split")) for p in ps]
for config_name, rows in groupby(exported, itemgetter("config"))
}
# default keeps only train-0001
```

## Proof (failing before fix)

```
FAILED - default only had 1 URL instead of 2
```

## Fix

Sort by `(config, split)` before both `groupby()` calls.

I'll follow up with a PR + regression test.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.