[Bug]: Performance degradation in Dataflow job when using `grpcio==1.45.0`
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Runner: Dataflow runner
SDK: Python
Version: 2.38.0
I recently swapped to using poetry for Python dependency management (and thus implicit deps have been inadvertently upgraded), and noticed a significant performance degradation with this new setup. After a lot of binary chopping, I have come to the conclusion that upgrading from `grpcio==1.44.0` to `grpcio==1.45.0` probably caused the degradation.
I don't have capacity to provide a reproducible example nor debug further, apologies, hopefully this is enough.
Here are some interesting screenshots:
Dataflow job CPU util pattern with `grpcio==1.44.0`

Dataflow job CPU util pattern with `grpcio==1.45.0`

Notice how the CPU utilisation is never capped at 100% in the second screenshot, they are both working on the exact same set of input data, exact same code except with `grpcio` and `grpcio-status` version upgraded.
### Issue Priority
Priority: 2
### Issue Component
Component: runner-dataflow
Contributor guide
Research direction
No source files or tests are named. Start by reproducing the reported Dataflow job with Python SDK 2.38.0, comparing grpcio and grpcio-status 1.44.0 against 1.45.0 under the same input and code. Done means confirming whether the dependency upgrade causes the reported CPU-utilization and performance difference, ideally with a reproducible case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100