[Bug]: Performance regression and stuck jobs due to gcs-connector v3 upgrade
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
#### Description
It has been reported that some pipelines are running slower after upgrading Beam SDK to 2.74+. Particularly, users observed increased CPU, memory usage, and wall time for those jobs.
#### Root Cause
We suspect the culprit is gcs-connector, since we bumped its version from 2.x to 3.x in release 2.74.
Specifically, in gcs-connector v3, `GoogleCloudStorageReadOptions.DEFAULT` changed its default fadvise behavior from `SEQUENTIAL` to `AUTO`. This change disrupts expected sequential read throughput and caching behavior, which leads to the observed performance degradation or even stuck jobs.
#### References
A separate investigation documented in the upstream Hadoop connectors repository has confirmed this behavior: [GoogleCloudDataproc/hadoop-connectors#1762](https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1762)
Internal bugs: 535192796, 512366613, 535194786.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with the Beam 2.74+ gcs-connector v3 upgrade and the upstream GoogleCloudDataproc/hadoop-connectors#1762 investigation, focusing on GoogleCloudStorageReadOptions.DEFAULT and its fadvise behavior. Reproduce or compare sequential-read pipelines before and after the upgrade; done means the reported CPU, memory, wall-time, and stuck-job regression is addressed and the behavior is validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100