apache / apache/arrow

[Python] pyarrow.concat_tables raises error about different Schema if columns have different order

Open
#35,424 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

why different columns order result in different schema?

the following code raise:

pyarrow.lib.ArrowInvalid: Schema at index 1 was different:

```python
import pandas as pd
import pyarrow as pa

print(pa.show_info())

df1 = pd.DataFrame({
"col1": [1,2,3,4,5],
"col2": ["A", "B", "C", "D", "E"],
})

df2 = pd.DataFrame({
"col2": ["A", "B", "C", "D", "E"],
"col1": [1,2,3,4,5],
})

tbl1 = pa.Table.from_pandas(df1, preserve_index=False)
tbl2 = pa.Table.from_pandas(df2, preserve_index=False)

tbl3 = pa.concat_tables([tbl1, tbl2])
```

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Research direction

Run the pandas-to-PyArrow reproducer and compare tbl1.schema with tbl2.schema before concat_tables. Then review the concat_tables schema requirements in the Python and C++ APIs. Done should clarify the expected behavior for differing column order and document or identify the supported resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.