[Python] pyarrow.concat_tables raises error about different Schema if columns have different order
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
why different columns order result in different schema?
the following code raise:
pyarrow.lib.ArrowInvalid: Schema at index 1 was different:
```python
import pandas as pd
import pyarrow as pa
print(pa.show_info())
df1 = pd.DataFrame({
"col1": [1,2,3,4,5],
"col2": ["A", "B", "C", "D", "E"],
})
df2 = pd.DataFrame({
"col2": ["A", "B", "C", "D", "E"],
"col1": [1,2,3,4,5],
})
tbl1 = pa.Table.from_pandas(df1, preserve_index=False)
tbl2 = pa.Table.from_pandas(df2, preserve_index=False)
tbl3 = pa.concat_tables([tbl1, tbl2])
```
### Component(s)
C++, Python
Contributor guide
Research direction
Run the pandas-to-PyArrow reproducer and compare tbl1.schema with tbl2.schema before concat_tables. Then review the concat_tables schema requirements in the Python and C++ APIs. Done should clarify the expected behavior for differing column order and document or identify the supported resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100