dask / dask/distributed

Data loss with `DataFrame.set_index(.., shuffle="disk")`

Open
#4,386 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
`DataFrame.set_index(..., shuffle="disk")` is loosing significant amount of data when multiple workers are used.

I.e. length of result dataframe is much smaller than length of initial dataframe.

**What you expected to happen**:

Length of dataframes before and after of `set_index` to be the same.

**Minimal Complete Verifiable Example**:

```python
cluster = dask_kubernetes.KubeCluster.from_yaml('worker.yaml', name=f'{os.getenv("HOSTNAME")}-dask', n_workers=6)
client = Client(cluster)

test_ddf = dd.from_pandas(pd.DataFrame({
'uuid': [str(uuid.uuid4()) for i in range(10000)],
}), chunksize=100)

len(test_ddf)
# 10000

len(test_ddf.set_index('uuid'))
# 10000

len(test_ddf.set_index('uuid', shuffle='disk'))
# 1669
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2.26.0
- Python version: 3.8.5.final.0
- Operating System: Linux
- Install method (conda, pip, source): conda

{'host': {'python': '3.8.5.final.0',
'python-bits': 64,
'OS': 'Linux',
'OS-release': '5.4.0-51-generic',
'machine': 'x86_64',
'processor': 'x86_64',
'byteorder': 'little',
'LC_ALL': 'None',
'LANG': 'None'},
'packages': {'python': '3.8.5.final.0',
'dask': '2.26.0',
'distributed': '2.26.0',
'msgpack': '1.0.0',
'cloudpickle': '1.6.0',
'tornado': '6.0.4',
'toolz': '0.10.0',
'numpy': '1.19.1',
'lz4': '3.1.0',
'blosc': None}}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.