apache / apache/paimon

[Bug] AppendTableSavepointTagFailoverITCase.testRegionFailoverPreservesSavepointTag is flaky

Đã đóng
#9,653 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Java
Star
3.4k
Fork
1.4k
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
396

Mô tả

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master, `475be566f` (2.1-SNAPSHOT).

### Compute Engine

Flink, `paimon-flink-common` integration tests.

### Minimal reproduce step

`AppendTableSavepointTagFailoverITCase.testRegionFailoverPreservesSavepointTag` fails intermittently in CI:

```
[ERROR] org.apache.paimon.flink.AppendTableSavepointTagFailoverITCase.testRegionFailoverPreservesSavepointTag Time elapsed: 0.837 s <<< ERROR!
java.util.concurrent.ExecutionException: org.apache.flink.runtime.checkpoint.CheckpointException:
Checkpoint triggering task Source: region-source (2/2) of job 361a330eeeffd33ab4ec1f2b89cc227c
is not being executed at the moment. Aborting checkpoint.
Failure reason: Not all required tasks are currently running.
at org.apache.paimon.flink.AppendTableSavepointTagFailoverITCase.testRegionFailoverPreservesSavepointTag(AppendTableSavepointTagFailoverITCase.java:262)
```

Two runs from today, on unrelated pull requests that touch neither Flink nor checkpointing:

- https://github.com/apache/paimon/actions/runs/33950240085/job/101263510564 (a `paimon-common` bitmap file index change)
- https://github.com/apache/paimon/actions/runs/33951866816/job/101268061352 (a `paimon-common` cast rule change)

The test injects a region failover, waits, and then triggers a second savepoint:

```java
// Wait until the job has recovered and resumed committing after the region failover.
waitUntilRecoveredAndCommitting(table);

client.triggerSavepoint(getTempDirPath("savepoint2_" + tableName), SavepointFormatType.DEFAULT)
.get(60, TimeUnit.SECONDS);
```

and the wait is defined as "a snapshot newer than the one we saw before":

```java
private void waitUntilRecoveredAndCommitting(FileStoreTable table) throws Exception {
Long baseline = table.snapshotManager().latestSnapshotId();
long base = baseline == null ? 0L : baseline;
long deadline = System.currentTimeMillis() + WAIT_TIMEOUT_MILLIS;
while (System.currentTimeMillis() < deadline) {
Long latest = table.snapshotManager().latestSnapshotId();
if (latest != null && latest > base) {
return;
}
Thread.sleep(200);
}
...
```

### What doesn't meet your expectations?

A new snapshot is not the same condition as "every task is running again", which is what `triggerSavepoint` requires. A commit that was already in flight when the failover started can bump the snapshot id and satisfy the wait while the failed region is still being redeployed, and the savepoint then aborts with the message above. The 0.837 s elapsed time says the savepoint failed almost immediately rather than timing out, which is consistent with that.

### Anything else?

Two ways to make it deterministic, whichever fits the suite better: wait on the job's own state (all vertices `RUNNING` via the cluster client) before triggering, or retry `triggerSavepoint` while it fails with a `CheckpointException` whose reason is that not all tasks are running, up to the existing timeout.

The test was added in #9309. I have not tried to reproduce it locally; the evidence is the two CI runs above, which failed at the same line with the same reason.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start with AppendTableSavepointTagFailoverITCase.testRegionFailoverPreservesSavepointTag at the reported line, then read waitUntilRecoveredAndCommitting and the region-failover setup. Run the integration test and inspect whether its recovery wait matches the condition required by triggerSavepoint. Done means the test reliably completes the second savepoint after failover without the not-all-tasks-running error.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
java
Lĩnh vực
distributed-systems, testing
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
70/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.