P2P shuffling failed during transfer when data includes mixed type
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
When there are object columns with mixed data types, the arrow backend cannot handle this.
The internal error that is raised in this example is
```python-traceback
ArrowInvalid: ('cannot mix list and non-list, non-null values', 'Conversion failed for column mixed_stuff with type object')
```
while the user receives a generic shuffle failed exception
```python-traceback
RuntimeError: P2P shuffling [id] failed during transfer phase
```
Reproducing code example
```python
import pandas as pd
import dask.dataframe as dd
import numpy as np
from distributed import Client
with Client() as client:
df = pd.DataFrame({
"mixed_stuff": [{"foo": "bar"}, np.array((3,))] * 2,
"int": [1, 2] * 2,
})
ddf = dd.from_pandas(df, npartitions=2)
ddf.shuffle(on="int").compute()
```
Contributor guide
Assessment
This issue has not been assessed yet.