apache / apache/arrow

[C++] Writing a dataset with 0 rows doesn't create any files

Open
#33,261 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

If the input data has no rows, no files get created. This is potentially unexpected as it looks like "nothing happened". It might be nicer to create an empty file. With partitioning, though, that then gets weird (there's no partition values) so maybe an error might make more sense instead.

Reproduction in Python
```python

import tempfile
from pathlib import Path

import pyarrow
import pyarrow.dataset

print("PyArrow version:", pyarrow.__version__)

table = pyarrow.table([
[],
], schema=pyarrow.schema([
("ints", "int64"),
]))

with tempfile.TemporaryDirectory() as d:
pyarrow.dataset.write_dataset(table, d, format="feather")
print(list(Path(d).iterdir()))
```
Output
```

> python repro.py
PyArrow version: 9.0.0
[]
```

**Reporter**: [David Li](https://issues.apache.org/jira/browse/ARROW-18060) / @lidavidm
#### Related issues:
- [[R] arrow::write_dataset() does nothing with 0 row dataframes in R](https://github.com/apache/arrow/issues/31933) (duplicates)

**Note**: *This issue was originally created as [ARROW-18060](https://issues.apache.org/jira/browse/ARROW-18060). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproduction using pyarrow.dataset.write_dataset and inspect the dataset-writing entry point involved in the empty-table case. Determine the expected behavior for zero rows, including partitioned datasets, and consider the issue complete when that behavior is implemented and verified against the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.