Flaky Test: KvSnapshotBatchScannerITCase.testKvSnapshotLease
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
Error: Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.961 s <<< FAILURE! - in org.apache.fluss.client.table.scanner.batch.KvSnapshotBatchScannerITCase
Error: org.apache.fluss.client.table.scanner.batch.KvSnapshotBatchScannerITCase.testKvSnapshotLease Time elapsed: 3.16 s <<< FAILURE!
org.opentest4j.AssertionFailedError:
expected: [1L, 1L, 1L]
but was: [1L, 0L, 1L]
The tests that were run was part of
https://github.com/apache/fluss/actions/runs/22756751008/job/66003123256
### Root Cause Analysis
Primary Race: Async triggerSnapshot in FlussClusterExtension
In FlussClusterExtension.triggerSnapshot() (line 747-775):
snapshotId = kvSnapshotManager.currentSnapshotId(); // read counter (e.g., 1)
kvSnapshotManager.triggerSnapshot(); // ASYNC - queues to guardedExecutor
nextSnapshotId = kvSnapshotManager.currentSnapshotId(); // read counter again
PeriodicSnapshotManager.triggerSnapshot() (line 186) submits work to guardedExecutor.execute(...) which is asynchronous. The actual initSnapshot() call — which increments snapshotIdCounter via
getAndIncrement() at KvTabletSnapshotTarget.java:183 — runs on the guardedExecutor thread, NOT the calling thread.
The race:
1. Test thread reads snapshotId = 1 (counter value after first snapshot)
2. Test thread calls triggerSnapshot() → queues async task
3. Test thread reads nextSnapshotId = 1 (counter hasn't been incremented yet)
4. nextSnapshotId == snapshotId → method returns null
5. triggerAndWaitSnapshots skips this bucket entirely
The async task eventually runs and creates snapshot 1, but the test has already moved on. When getLatestKvSnapshots() is called, this bucket may still show snapshot 0.
Secondary Race: Log offset not yet flushed to KV
In KvTabletSnapshotTarget.initSnapshot() (line 170-203), there's a check:
if (logOffset <= logOffsetOfLatestSnapshot) {
return Optional.empty(); // skip — no new data
}
Even if the async trigger runs, if the KV tablet hasn't applied the new log entries yet (from putRows), the snapshot is skipped because the flushed log offset hasn't advanced.
Flow of the failure
1. triggerAndWaitSnapshots → for bucket 1, triggerSnapshot returns null (race) → bucket 1 is skipped
2. Test calls admin.getLatestKvSnapshots() → bucket 1 still at snapshot 0
3. acquireSnapshots stores {bucket0: 1, bucket1: 0, bucket2: 1}
4. checkKvSnapshotLeaseEquals expects [1L, 1L, 1L] but gets [1L, 0L, 1L]
### Fix
Modify FlussClusterExtension.triggerSnapshot() to properly handle the async nature. Instead of checking currentSnapshotId() immediately after the async trigger, use a retry/poll approach to wait for
the snapshot ID to increment.
File to modify
- fluss-server/src/test/java/org/apache/fluss/server/testutils/FlussClusterExtension.java (lines 747-775)
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in fluss-server/src/test/java/org/apache/fluss/server/testutils/FlussClusterExtension.java, focusing on triggerSnapshot() around lines 747-775 and its asynchronous snapshot trigger. Reproduce the failure in KvSnapshotBatchScannerITCase.testKvSnapshotLease and verify that snapshot triggering waits for the snapshot ID to advance before checking the latest snapshots.】【。}સ аптоном? Wait accidental invalid. Need regenerate clean. string can mention test no command. Need exact JSON no weird. Ensure 2 sentences. en. Let's final._modifier? We need output only. figure score maybe 55. Why malformed previous not sent? final now. [[
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100