apache / apache/doris

[Bug] CCR IngestBinlog may commit MOW rowsets after delete bitmap calculation fails

Open
#67,120 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

- 3.1.4

### What's Wrong?

CCR incremental replication applies upstream rowsets on the target cluster through `IngestBinlog`. For a UNIQUE KEY merge-on-write table, the Shared-Nothing ingest path submits commit-phase delete bitmap work and waits for its asynchronous token, but discards both returned statuses:

```cpp
static_cast(BaseTablet::commit_phase_update_delete_bitmap(...));
static_cast(calc_delete_bitmap_token->wait());
```

If delete bitmap calculation fails—for example, because an allocation is rejected by the system-memory low-water-mark check—the worker logs the error, but `IngestBinlog` can continue to commit the rowset. The version can then be published without the required delete bitmap.

MOW reads rely on the persisted delete bitmap instead of performing read-time key merging. Consequently, historical rows for the same unique key can become visible, producing duplicate unique keys with different values or update timestamps.

This issue is specific to the Shared-Nothing CCR `IngestBinlog` path. Ordinary load paths use different transaction and publish-version error propagation.

There is also a related cleanup gap: the deferred failure handler captured `download_success_files` by value before downloads occurred, so files downloaded by a failed ingest were not included in cleanup.

### What You Expected?

Any synchronous submission failure or asynchronous delete bitmap calculation failure must:

1. Be propagated through the `IngestBinlog` response.
2. Abort the local tablet transaction before rowset commit/publish.
3. Leave the failed version invisible.
4. Remove files downloaded by the failed ingest.
5. Allow CCR to retry the same binlog safely.

No rowset should become visible unless all required MOW delete bitmap calculations have completed successfully.

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at the Shared-Nothing CCR IngestBinlog path and trace BaseTablet::commit_phase_update_delete_bitmap(...) and calc_delete_bitmap_token->wait(). Check how their statuses reach the response and rowset commit, and how download_success_files is captured for cleanup. Done means failures abort the transaction, keep the version invisible, clean up downloaded files, and permit a safe retry.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.