huggingface / huggingface/datasets
concatenate_datasets(axis=1) drops the columns of a source with no rows instead of raising
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
`concatenate_datasets([ds1, ds2], axis=1)` returns a dataset without `ds2`'s columns when `ds2` has 0 rows, and raises nothing. The row-count check that rejects every other length mismatch never runs.
### Steps to reproduce
```python
from datasets import Dataset, Features, Value, concatenate_datasets
ds1 = Dataset.from_dict({"col_1": [0, 1, 2]})
ds2 = Dataset.from_dict({"col_2": []}, features=Features({"col_2": Value("int64")}))
concatenate_datasets([ds1, ds2], axis=1).column_names
# ['col_1'] (col_2 is gone)
```
A 2-row `ds2` raises instead:
```python
ds2 = Dataset.from_dict({"col_2": [0, 1]})
concatenate_datasets([ds1, ds2], axis=1)
# ValueError: Number of rows must match for all datasets
```
When every source has 0 rows, the result keeps only the first source's columns:
```python
a = Dataset.from_dict({"col_1": []}, features=Features({"col_1": Value("int64")}))
b = Dataset.from_dict({"col_2": []}, features=Features({"col_2": Value("string")}))
concatenate_datasets([a, b], axis=1).column_names
# ['col_1'] (expected ['col_1', 'col_2'])
```
`.filter()` and `.select([])` produce 0-row sources that hit the same path, so a per-column computation which filters to nothing drops its column instead of failing.
### Cause
`_concatenate_map_style_datasets` removes rowless sources before dispatching on `axis` (src/datasets/arrow_dataset.py:7057):
```python
# Ignore datasets with no rows
if any(dset.num_rows > 0 for dset in dsets):
dsets = [dset for dset in dsets if dset.num_rows > 0]
else:
# Return first dataset if all datasets are empty
return dsets[0]
# Perform checks (and a potential cast if axis=0)
if axis == 0:
_check_if_features_can_be_aligned([dset.features for dset in dsets])
else:
if not all(dset.num_rows == dsets[0].num_rows for dset in dsets):
raise ValueError("Number of rows must match for all datasets")
```
Dropping a rowless source is correct for `axis=0`, where it contributes no rows. On `axis=1` it contributes columns, so removing it deletes those columns and hides the length mismatch from the check below it.
The check ran unconditionally before #3195:
```python
elif axis == 1 and not all([dset.num_rows == dsets[0].num_rows for dset in dsets]):
raise ValueError("Number of rows must match for all datasets")
```
### Environment
`datasets` 5.0.2.dev0 installed from source at 19f69de, Python 3.13.12, pyarrow 25.0.1, macOS arm64.
### One question about axis=0
`concatenate_datasets([ds1, ds2], axis=0)` with a rowless `ds2` also drops `col_2`, where `test_concatenate_datasets_new_columns` shows a non-empty `ds2` would have added it filled with `None`. Same root cause, but "ignore rowless sources" reads deliberate for `axis=0`, so I left that behavior alone. Tell me if you want the column kept there too and I'll extend the fix.
### Not the same as #8341
#8341 is the `IterableDataset` Arrow fast path, where `HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrow` appends onto a leaked loop variable instead of its accumulator. This one is map-style `_concatenate_map_style_datasets`, and the columns go missing because the rowless source is filtered out before the `axis` dispatch. Different file, different mechanism.
Contributor guide
Research direction
Start in src/datasets/arrow_dataset.py at _concatenate_map_style_datasets, around line 7057, and inspect the existing test_concatenate_datasets_new_columns coverage. Add regression coverage for axis=1 with rowless sources, then verify that row-count mismatches raise and that columns from empty sources are not silently dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100