dask / dask/dask

Documentation for `set_index(col, compute=True)` is unclear/inaccurate

Open
#8,415 13 comments 0 reactions 0 assignees View on GitHub
dataframe documentation needs attention
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

I think the documentation is currently unclear/inaccurate about the nature of the `compute` parameter for `set_index`:

```python
df.set_index(col, compute=True)
```

The documentation currently contains this description:
> compute: bool, default False
> - Whether or not to trigger an immediate computation. Defaults to False. Note, that even if you set compute=False, an immediate computation will still be triggered if divisions is None.

This would suggest that if I provide divisions and set `compute=True`, immediate computation will be triggered. This only seems to be the case when using `shuffle=disk`, however. Even then, it's not clear to me what is actually being computed.

Examples from the SO question where I originally asked about this ([What does set_index(col, compute=True) do in Dask?](https://stackoverflow.com/questions/70012274/what-does-set-indexcol-compute-true-do-in-dask)):

```python
import dask.datasets
df = dask.datasets.timeseries()

# Nothing gets submitted to the scheduler
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
compute=True
)
```

Going down the stack of functions `set_index` actually calls, it appears that the only place where `compute` is actually used in [`rearrange_by_column_disk`](https://github.com/dask/dask/blob/8aea537d925b794a94f828d35211a5da05ad9dce/dask/dataframe/shuffle.py#L501). And indeed:

```python
# Still, nothing gets submitted
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
shuffle='tasks',
compute=True
)

# Something is computed here
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
shuffle='disk',
compute=True
)
```

If I'm correct, then I believe the documentation should reflect the fact that this setting only affects the `shuffle=disk` case. Also, I can't work out from the documentation what is actually being computed — "immediate computation" of what?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.