DataFrame to Arrow Table Conversion issue with Tuple datatype
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
While attempting to convert a DataFrame containing a column with tuples to an Arrow Table and back to a DataFrame, we encountered a discrepancy (expected tuple, got np.ndarray) between the original DataFrame and the DataFrame obtained from the Arrow Table.
Steps to Reproduce:
1. Create a DataFrame with a column that contains tuples.
2. Convert the DataFrame to an Arrow Table using pa.Table.from_pandas().
3. Convert the Arrow Table back to a DataFrame using table.to_pandas().
4. Compare the original DataFrame with the DataFrame obtained from the Arrow Table.
Expected Results:
The DataFrames before and after conversion should be identical.
Actual Results:
The DataFrames differ.
```
import pandas as pd
import pyarrow as pa
data = {'tup': [(1, 2), (3, 4), (5, 6)]}
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df)
df_back = table.to_pandas()
print("Original DataFrame:")
print(df)
print("\nDataFrame converted from Arrow Table:")
print(df_back)
print("\nAre the DataFrames equal?", df.equals(df_back))
```

### Component(s)
Python
Contributor guide
Research direction
Start with the reproducer in the issue using pandas and pyarrow, then inspect the Python component responsible for pa.Table.from_pandas() and table.to_pandas(). Verify the tuple column's values before and after the round trip; done means the resulting DataFrame matches the original and df.equals(df_back) succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100