Documentation for `set_index(col, compute=True)` is unclear/inaccurate
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
I think the documentation is currently unclear/inaccurate about the nature of the `compute` parameter for `set_index`:
```python
df.set_index(col, compute=True)
```
The documentation currently contains this description:
> compute: bool, default False
> - Whether or not to trigger an immediate computation. Defaults to False. Note, that even if you set compute=False, an immediate computation will still be triggered if divisions is None.
This would suggest that if I provide divisions and set `compute=True`, immediate computation will be triggered. This only seems to be the case when using `shuffle=disk`, however. Even then, it's not clear to me what is actually being computed.
Examples from the SO question where I originally asked about this ([What does set_index(col, compute=True) do in Dask?](https://stackoverflow.com/questions/70012274/what-does-set-indexcol-compute-true-do-in-dask)):
```python
import dask.datasets
df = dask.datasets.timeseries()
# Nothing gets submitted to the scheduler
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
compute=True
)
```
Going down the stack of functions `set_index` actually calls, it appears that the only place where `compute` is actually used in [`rearrange_by_column_disk`](https://github.com/dask/dask/blob/8aea537d925b794a94f828d35211a5da05ad9dce/dask/dataframe/shuffle.py#L501). And indeed:
```python
# Still, nothing gets submitted
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
shuffle='tasks',
compute=True
)
# Something is computed here
df.set_index(
'name',
divisions=('Alice', 'Michael', 'Zelda'),
shuffle='disk',
compute=True
)
```
If I'm correct, then I believe the documentation should reflect the fact that this setting only affects the `shuffle=disk` case. Also, I can't work out from the documentation what is actually being computed — "immediate computation" of what?
Contributor guide
Assessment
This issue has not been assessed yet.