Data loss with `DataFrame.set_index(.., shuffle="disk")`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
`DataFrame.set_index(..., shuffle="disk")` is loosing significant amount of data when multiple workers are used.
I.e. length of result dataframe is much smaller than length of initial dataframe.
**What you expected to happen**:
Length of dataframes before and after of `set_index` to be the same.
**Minimal Complete Verifiable Example**:
```python
cluster = dask_kubernetes.KubeCluster.from_yaml('worker.yaml', name=f'{os.getenv("HOSTNAME")}-dask', n_workers=6)
client = Client(cluster)
test_ddf = dd.from_pandas(pd.DataFrame({
'uuid': [str(uuid.uuid4()) for i in range(10000)],
}), chunksize=100)
len(test_ddf)
# 10000
len(test_ddf.set_index('uuid'))
# 10000
len(test_ddf.set_index('uuid', shuffle='disk'))
# 1669
```
**Anything else we need to know?**:
**Environment**:
- Dask version: 2.26.0
- Python version: 3.8.5.final.0
- Operating System: Linux
- Install method (conda, pip, source): conda
{'host': {'python': '3.8.5.final.0',
'python-bits': 64,
'OS': 'Linux',
'OS-release': '5.4.0-51-generic',
'machine': 'x86_64',
'processor': 'x86_64',
'byteorder': 'little',
'LC_ALL': 'None',
'LANG': 'None'},
'packages': {'python': '3.8.5.final.0',
'dask': '2.26.0',
'distributed': '2.26.0',
'msgpack': '1.0.0',
'cloudpickle': '1.6.0',
'tornado': '6.0.4',
'toolz': '0.10.0',
'numpy': '1.19.1',
'lz4': '3.1.0',
'blosc': None}}
Contributor guide
Assessment
This issue has not been assessed yet.