[Python] pyarrow.dataset.write_dataset doesn't write empty datasets, even if they had a schema, thus losing it when reading back the dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
The write_dataset function doesn't write anything if the table is empty. This has the consequence that schema information is lost when reading it back (because it didn't write anything in the first place).
This can be avoided by manually writing an empty table with that schema inside the destination folder in the parquet format. But I expected this to already be the intended behaviour.
To reproduce:
```python
import os
import shutil
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as pds
dest = tempfile.mkdtemp(suffix='.parquet')
schema = pa.schema([('a', pa.int64()), ('b', pa.int64())])
empty_table = schema.empty_table()
pds.write_dataset(empty_table, dest, format='parquet')
ds = pds.dataset(dest)
print(ds.schema)
# -> no output
# I expected ds.schema == pa.schema([('a', pa.int64()), ('b', pa.int64())])
print(os.listdir(dest))
# -> []
# I expected to find an empty parquet file in the folder to avoid losing the schema!
# WORKAROUND
# If I write the empty table with pyarrow.parquet.write_table inside the folder, then the schema is preserved
pq.write_table(empty_table, os.path.join(dest, 'part-0.parquet'))
ds_with_workaround = pds.dataset(dest)
assert ds_with_workaround.schema == schema
# delete folder recursively
shutil.rmtree(dest, ignore_errors=True)
```
Thank you
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.