apache / apache/arrow

[C++][Python] pyarrow.dataset.write_dataset does not store fields metadata

Open
#34,067 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python good-second-issue Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hello,
I noticed that `pyarrow.dataset.write_dataset` does not store field metadata. Could you please, tell me if it is expected behavior?

I am using pyarrow version 10.0.1. I also tried 11.0.0, and the behavior is the same.

```
import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
'n_legs': [2, 4, 5, 100],
'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
schema = pa.schema([
pa.field('year', pa.int16(), metadata={"123": b"123"}),
pa.field('n_legs', pa.int64()),
pa.field('animals', pa.string())],
metadata={"xyz": b"xyz"})
t = pa.Table.from_pandas(df, schema)
print(t.field(0).metadata)
# prints {b'123': b'123'}

pyarrow.dataset.write_dataset(
data=t,
base_dir="xyz",
schema=t.schema,
format="arrow",
)
```
Then load the stored file

```
with pa.memory_map("xyz/part-0.arrow", "r") as source:
table = pa.ipc.open_file(source).read_all()
print(table.field(0).metadata)
# prints None
print(table.schema.metadata)
# prints {b'xyz': b'xyz', b'pandas': ... }
```

As you can see above, schema metadata are preserved.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided Python example, starting at the pyarrow.dataset.write_dataset entry point and checking the stored file with pa.ipc.open_file. Done means field metadata is preserved in the Arrow file as schema metadata already is, with a regression test covering the reported schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.