Reshuffle causes all work to be done by one machine
- Dominant language
- Go
- Stars
- 11
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I've got a pipeline that looks like this:

I start the job with 20 workers, and it's able to read from GCS pretty quickly. However, the `Reshuffle` steps seems to all get sent to a single worker ( assuming this because Dataflow scales down to a single worker while `Reshuffle` is doing it's thing ).
I suspect it's because of the `GroupByKey`:

Which seems to be processing all of the elements at once.
Is there a way to prevent this so that this step doesn't cause Dataflow to scale down a bunch of workers and then scale back up once it's done? Maybe assign a random time between `time.Now()` and `time.Now() + 1 hour`, window into 1 minute buckets, then put everything back in the global window before returning the PCollection?
Or is there something else I should be doing later in the pipeline to ensure that `Reshuffle` doesn't all get done on a single worker?
Contributor guide
No contributing guide indexed for this repository
Research direction
The payload names no repository files or tests. Start by reproducing the Go pipeline on Dataflow and inspecting the Reshuffle and GroupByKey stages; done means establishing whether the single-worker behavior is expected and documenting or implementing a supported way to avoid the scale-down.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100