apache / apache/beam

[Bug]: BigtableIO.readChangeStream() with RESUME_OR_NEW can start duplicate consumers inside one Dataflow job

Open
#39,970 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage bug io P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

## Environment

- Apache Beam Java SDK 2.75.0
- Google Cloud Dataflow / Portable Runner
- Exactly one active job and one worker
- Stable, explicitly configured `changeStreamName`

## Configuration

```java
BigtableIO.readChangeStream()
.withProjectId(projectId)
.withInstanceId(instanceId)
.withTableId(tableId)
.withAppProfileId(appProfileId)
.withMetadataTableTableId(metadataTableId)
.withChangeStreamName(changeStreamName)
.withExistingPipelineOptions(
BigtableIO.ExistingPipelineOptions.RESUME_OR_NEW);
```

## Observed behavior

1. The job starts normally and consumes every event once.
2. After approximately 30 minutes, the running job logs:

```text
Resuming from previous pipeline with low watermark of ...
```

3. Every subsequent event is then consumed exactly twice.
4. No second job is started and the worker count remains one.
5. The duplicate consumption is persistent, not just a short overlap during a checkpoint.
6. Omitting `withExistingPipelineOptions(...)`, thereby using the default `FAIL_IF_EXISTS`, prevents the problem.

## Expected behavior

Retries, checkpointing, or repeated initialization within the same active job must not cause it to resume itself or create another consumer for partitions already being consumed.

## Suspected cause

Dataflow can re-execute `InitializeDoFn`. After the current job has created its metadata, `RESUME_OR_NEW` appears to interpret that metadata as belonging to a previous pipeline. The observed `Resuming from previous pipeline...` log line indicates that this branch is executed while the original consumer is still active.

`ResumeFromPreviousPipelineAction` restores the existing partition UUIDs. The locking code in `MetadataTableDao.lockAndRecordPartition()` considers an existing lock with the same UUID successfully held:

```java
if (doHoldLock(
partitionRecord.getPartition(),
partitionRecord.getUuid())) {
return true;
}
```

The original and resumed readers can therefore both pass the lock check and open change-stream RPCs for the same partitions.

The workaround supports this explanation. With the default `FAIL_IF_EXISTS`, repeated initialization detects the existing metadata and returns without emitting another `InitialPipelineState`, preventing another downstream consumer chain from starting.

## Workaround

Omit `withExistingPipelineOptions(...)`. This prevents duplicate initialization, but it also prevents a legitimate later job from resuming existing metadata, so it is not a complete solution.

## Suggested fixes

1. Make initialization idempotent per job execution. Store a unique execution identifier in the metadata and claim initialization atomically. Re-execution with the same identifier should emit nothing, while a genuinely new job should still be allowed to resume.

2. Add a durable checkpoint or reshuffle boundary between `InitializeDoFn` and the unbounded `DetectNewPartitionsDoFn`. This should ensure that retries or backup executions of initialization cannot start multiple downstream consumer chains.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start by tracing InitializeDoFn and ResumeFromPreviousPipelineAction, then inspect MetadataTableDao.lockAndRecordPartition() and the RESUME_OR_NEW initialization path. Reproduce the repeated-initialization behavior with one active Dataflow job and verify that retries do not create duplicate consumers, while a genuinely new job can still resume existing metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, java
Domain
cloud, data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.