dask / dask/dask

Unexpected behavior with Pyarrow string conversion

Open
#10,546 1 comment 0 reactions 0 assignees View on GitHub
bug convert-string dataframe
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

**Minimal Complete Verifiable Example**:

```python
# Put your MCVE code here
import pandas as pd
import dask.dataframe as dd

df = dd.from_pandas(pd.DataFrame({"col1": ["a","a","a", "b", "c", "c"], "col2": [1,2,3,4,5,6], "col3": [43,23,12,32,65,76]}), npartitions=2)
df.groupby("col1").agg({"col2": list, "col3": "first"}).reset_index().compute().info()
```

**Anything else we need to know?**:
In this example, the column 'col2' is being converted to a string instead of an array (object), which makes future processing more difficult. Is there a way to ignore certain columns from auto string conversion? I am aware of the configuration setting dataframe.convert-string, but it applies to all columns.
##### Dataframe
image

##### Dataframe after groupby
image
image

**Environment**:

- Dask version: 2023.9.3
- Python version: 3.9.15
- Operating System: Centos 7
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by running the provided MCVE with the reported Dask 2023.9.3 and Python 3.9.15 environment, compare col2 before and after the groupby aggregation, and consider the issue done when list-valued output is preserved or the supported per-column conversion behavior is clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.