apache / apache/arrow

DataFrame to Arrow Table Conversion issue with Tuple datatype

Open
#38,344 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

While attempting to convert a DataFrame containing a column with tuples to an Arrow Table and back to a DataFrame, we encountered a discrepancy (expected tuple, got np.ndarray) between the original DataFrame and the DataFrame obtained from the Arrow Table.

Steps to Reproduce:

1. Create a DataFrame with a column that contains tuples.
2. Convert the DataFrame to an Arrow Table using pa.Table.from_pandas().
3. Convert the Arrow Table back to a DataFrame using table.to_pandas().
4. Compare the original DataFrame with the DataFrame obtained from the Arrow Table.

Expected Results:

The DataFrames before and after conversion should be identical.

Actual Results:

The DataFrames differ.

```

import pandas as pd
import pyarrow as pa

data = {'tup': [(1, 2), (3, 4), (5, 6)]}
df = pd.DataFrame(data)

table = pa.Table.from_pandas(df)
df_back = table.to_pandas()

print("Original DataFrame:")
print(df)

print("\nDataFrame converted from Arrow Table:")
print(df_back)

print("\nAre the DataFrames equal?", df.equals(df_back))

```

![image](https://github.com/apache/arrow/assets/24271600/ed9ccf51-23f4-4fbb-bab8-c652245c8455)

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the reproducer in the issue using pandas and pyarrow, then inspect the Python component responsible for pa.Table.from_pandas() and table.to_pandas(). Verify the tuple column's values before and after the round trip; done means the resulting DataFrame matches the original and df.equals(df_back) succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.