dask / dask/distributed

[P2P] `merge_unpack` does not reschedule on `ShuffleClosedError`

Open
#8,660 0 comments 0 reactions 0 assignees View on GitHub
bug shuffle
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I just noticed that `merge_unpack` (https://github.com/hendrikmakait/distributed/blob/9fae5dacf4d2cfd5c659e472b0a3ef307d695863/distributed/shuffle/_merge.py#L163-L196) isn't wrapped in `handle_unpack_errors` like `shuffle_unpack` (https://github.com/hendrikmakait/distributed/blob/9fae5dacf4d2cfd5c659e472b0a3ef307d695863/distributed/shuffle/_shuffle.py#L94-L101).

This could cause P2P merging not to restart in some edge cases. (I'm not sure which ones, but there should be tests for P2P shuffling that will start to fail if we remove `handle_unpack_errors`.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.