apache / apache/beam

[Bug]: Performance regression and stuck jobs due to gcs-connector v3 upgrade

Open
#39,548 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage bug java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

#### Description
It has been reported that some pipelines are running slower after upgrading Beam SDK to 2.74+. Particularly, users observed increased CPU, memory usage, and wall time for those jobs.

#### Root Cause
We suspect the culprit is gcs-connector, since we bumped its version from 2.x to 3.x in release 2.74.

Specifically, in gcs-connector v3, `GoogleCloudStorageReadOptions.DEFAULT` changed its default fadvise behavior from `SEQUENTIAL` to `AUTO`. This change disrupts expected sequential read throughput and caching behavior, which leads to the observed performance degradation or even stuck jobs.

#### References
A separate investigation documented in the upstream Hadoop connectors repository has confirmed this behavior: [GoogleCloudDataproc/hadoop-connectors#1762](https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1762)
Internal bugs: 535192796, 512366613, 535194786.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start with the Beam 2.74+ gcs-connector v3 upgrade and the upstream GoogleCloudDataproc/hadoop-connectors#1762 investigation, focusing on GoogleCloudStorageReadOptions.DEFAULT and its fadvise behavior. Reproduce or compare sequential-read pipelines before and after the upgrade; done means the reported CPU, memory, wall-time, and stuck-job regression is addressed and the behavior is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.