dask / dask/distributed

P2P shuffling failed during transfer when data includes mixed type

Open
#8,310 1 comment 0 reactions 0 assignees View on GitHub
bug shuffle
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When there are object columns with mixed data types, the arrow backend cannot handle this.

The internal error that is raised in this example is

```python-traceback
ArrowInvalid: ('cannot mix list and non-list, non-null values', 'Conversion failed for column mixed_stuff with type object')
```

while the user receives a generic shuffle failed exception

```python-traceback
RuntimeError: P2P shuffling [id] failed during transfer phase
```

Reproducing code example

```python
import pandas as pd
import dask.dataframe as dd
import numpy as np
from distributed import Client
with Client() as client:

df = pd.DataFrame({
"mixed_stuff": [{"foo": "bar"}, np.array((3,))] * 2,
"int": [1, 2] * 2,
})
ddf = dd.from_pandas(df, npartitions=2)
ddf.shuffle(on="int").compute()
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.