apache / apache/beam

[Bug]: Performance degradation in Dataflow job when using `grpcio==1.45.0`

Open
#22,159 5 comments 0 reactions 0 assignees View on GitHub
bug dataflow P2 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Runner: Dataflow runner
SDK: Python
Version: 2.38.0

I recently swapped to using poetry for Python dependency management (and thus implicit deps have been inadvertently upgraded), and noticed a significant performance degradation with this new setup. After a lot of binary chopping, I have come to the conclusion that upgrading from `grpcio==1.44.0` to `grpcio==1.45.0` probably caused the degradation.

I don't have capacity to provide a reproducible example nor debug further, apologies, hopefully this is enough.

Here are some interesting screenshots:

Dataflow job CPU util pattern with `grpcio==1.44.0`
![Selection_502](https://user-images.githubusercontent.com/8895126/177336591-1aa66411-97e7-4511-9a96-9db43de89efb.png)

Dataflow job CPU util pattern with `grpcio==1.45.0`
![Selection_500](https://user-images.githubusercontent.com/8895126/177336610-698b866a-a14b-4300-951e-b6c21f98194e.png)

Notice how the CPU utilisation is never capped at 100% in the second screenshot, they are both working on the exact same set of input data, exact same code except with `grpcio` and `grpcio-status` version upgraded.

### Issue Priority

Priority: 2

### Issue Component

Component: runner-dataflow

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the reported Dataflow job with Python SDK 2.38.0, comparing grpcio and grpcio-status 1.44.0 against 1.45.0 under the same input and code. Done means confirming whether the dependency upgrade causes the reported CPU-utilization and performance difference, ideally with a reproducible case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.