[C++][Python] pyarrow.dataset.write_dataset does not store fields metadata
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
Hello,
I noticed that `pyarrow.dataset.write_dataset` does not store field metadata. Could you please, tell me if it is expected behavior?
I am using pyarrow version 10.0.1. I also tried 11.0.0, and the behavior is the same.
```
import pandas as pd
import pyarrow as pa
import pyarrow.dataset
df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
'n_legs': [2, 4, 5, 100],
'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
schema = pa.schema([
pa.field('year', pa.int16(), metadata={"123": b"123"}),
pa.field('n_legs', pa.int64()),
pa.field('animals', pa.string())],
metadata={"xyz": b"xyz"})
t = pa.Table.from_pandas(df, schema)
print(t.field(0).metadata)
# prints {b'123': b'123'}
pyarrow.dataset.write_dataset(
data=t,
base_dir="xyz",
schema=t.schema,
format="arrow",
)
```
Then load the stored file
```
with pa.memory_map("xyz/part-0.arrow", "r") as source:
table = pa.ipc.open_file(source).read_all()
print(table.field(0).metadata)
# prints None
print(table.schema.metadata)
# prints {b'xyz': b'xyz', b'pandas': ... }
```
As you can see above, schema metadata are preserved.
### Component(s)
Python
Contributor guide
Research direction
Reproduce the issue with the provided Python example, starting at the pyarrow.dataset.write_dataset entry point and checking the stored file with pa.ipc.open_file. Done means field metadata is preserved in the Arrow file as schema metadata already is, with a regression test covering the reported schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100