huggingface / huggingface/datasets

concatenate_datasets(axis=1) drops the columns of a source with no rows instead of raising

Open
#8,572 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

`concatenate_datasets([ds1, ds2], axis=1)` returns a dataset without `ds2`'s columns when `ds2` has 0 rows, and raises nothing. The row-count check that rejects every other length mismatch never runs.

### Steps to reproduce

```python
from datasets import Dataset, Features, Value, concatenate_datasets

ds1 = Dataset.from_dict({"col_1": [0, 1, 2]})
ds2 = Dataset.from_dict({"col_2": []}, features=Features({"col_2": Value("int64")}))

concatenate_datasets([ds1, ds2], axis=1).column_names
# ['col_1'] (col_2 is gone)
```

A 2-row `ds2` raises instead:

```python
ds2 = Dataset.from_dict({"col_2": [0, 1]})
concatenate_datasets([ds1, ds2], axis=1)
# ValueError: Number of rows must match for all datasets
```

When every source has 0 rows, the result keeps only the first source's columns:

```python
a = Dataset.from_dict({"col_1": []}, features=Features({"col_1": Value("int64")}))
b = Dataset.from_dict({"col_2": []}, features=Features({"col_2": Value("string")}))

concatenate_datasets([a, b], axis=1).column_names
# ['col_1'] (expected ['col_1', 'col_2'])
```

`.filter()` and `.select([])` produce 0-row sources that hit the same path, so a per-column computation which filters to nothing drops its column instead of failing.

### Cause

`_concatenate_map_style_datasets` removes rowless sources before dispatching on `axis` (src/datasets/arrow_dataset.py:7057):

```python
# Ignore datasets with no rows
if any(dset.num_rows > 0 for dset in dsets):
dsets = [dset for dset in dsets if dset.num_rows > 0]
else:
# Return first dataset if all datasets are empty
return dsets[0]

# Perform checks (and a potential cast if axis=0)
if axis == 0:
_check_if_features_can_be_aligned([dset.features for dset in dsets])
else:
if not all(dset.num_rows == dsets[0].num_rows for dset in dsets):
raise ValueError("Number of rows must match for all datasets")
```

Dropping a rowless source is correct for `axis=0`, where it contributes no rows. On `axis=1` it contributes columns, so removing it deletes those columns and hides the length mismatch from the check below it.

The check ran unconditionally before #3195:

```python
elif axis == 1 and not all([dset.num_rows == dsets[0].num_rows for dset in dsets]):
raise ValueError("Number of rows must match for all datasets")
```

### Environment

`datasets` 5.0.2.dev0 installed from source at 19f69de, Python 3.13.12, pyarrow 25.0.1, macOS arm64.

### One question about axis=0

`concatenate_datasets([ds1, ds2], axis=0)` with a rowless `ds2` also drops `col_2`, where `test_concatenate_datasets_new_columns` shows a non-empty `ds2` would have added it filled with `None`. Same root cause, but "ignore rowless sources" reads deliberate for `axis=0`, so I left that behavior alone. Tell me if you want the column kept there too and I'll extend the fix.

### Not the same as #8341

#8341 is the `IterableDataset` Arrow fast path, where `HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrow` appends onto a leaked loop variable instead of its accumulator. This one is map-style `_concatenate_map_style_datasets`, and the columns go missing because the rowless source is filtered out before the `axis` dispatch. Different file, different mechanism.

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/arrow_dataset.py at _concatenate_map_style_datasets, around line 7057, and inspect the existing test_concatenate_datasets_new_columns coverage. Add regression coverage for axis=1 with rowless sources, then verify that row-count mismatches raise and that columns from empty sources are not silently dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.