bramp / bramp/morebeam

Reshuffle causes all work to be done by one machine

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I've got a pipeline that looks like this:

![image](https://user-images.githubusercontent.com/444767/55018902-0ef12080-4fb1-11e9-9641-dd3c3854c26c.png)

I start the job with 20 workers, and it's able to read from GCS pretty quickly. However, the `Reshuffle` steps seems to all get sent to a single worker ( assuming this because Dataflow scales down to a single worker while `Reshuffle` is doing it's thing ).

I suspect it's because of the `GroupByKey`:

![image](https://user-images.githubusercontent.com/444767/55018999-43fd7300-4fb1-11e9-9acc-6349ed473276.png)

Which seems to be processing all of the elements at once.

Is there a way to prevent this so that this step doesn't cause Dataflow to scale down a bunch of workers and then scale back up once it's done? Maybe assign a random time between `time.Now()` and `time.Now() + 1 hour`, window into 1 minute buckets, then put everything back in the global window before returning the PCollection?

Or is there something else I should be doing later in the pipeline to ensure that `Reshuffle` doesn't all get done on a single worker?

Contributor guide

No contributing guide indexed for this repository

Research direction

The payload names no repository files or tests. Start by reproducing the Go pipeline on Dataflow and inspecting the Reshuffle and GroupByKey stages; done means establishing whether the single-worker behavior is expected and documenting or implementing a supported way to avoid the scale-down.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.