[test] ReplicaTest is flaky due to stale gauges in shared test metric groups
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
### Problem
`ReplicaTest` may read a stale `physicalStorage/localLogSize` gauge left by another test running in the same reused Maven Surefire JVM.
The following assertions can fail:
```text
ReplicaTest.testBucketPhysicalStorageLocalLogSizeIncludesFollower
expected: 450L
but was: 10830L
ReplicaTest.testPhysicalStorageLocalLogSizeIsScopedPerBucket
expected: 450L
but was: 10830L
```
Running `ReplicaTest` alone, for example directly from IntelliJ IDEA, normally succeeds. The failure only occurs when another test using the same table and bucket registers the metric first in the same reused Surefire fork.
The default test configuration uses two Surefire forks with fork reuse enabled. Therefore, whether the affected tests fail depends on test-class assignment and execution order, making the CI failure flaky.
### Expected behavior
Each test should use metrics associated with the `LogTablet` created by that test. The local log size gauge should return `450L`.
### Actual behavior
The test may reuse a bucket metric group created by an earlier test and read its stale gauge. In the observed failure, the gauge still references a previous `LogTablet` whose size is `10830L`.
### Root cause
`ReplicaTestBase` uses the static singleton instances:
```java
TestingMetricGroups.TABLET_SERVER_METRICS
TestingMetricGroups.USER_METRICS
```
These objects are shared by all tests executed in the same JVM.
`TabletServerMetricGroup` and `TableMetricGroup` cache metric groups using `computeIfAbsent`. When another replica for the same table and bucket registers the same gauge, the existing metric is retained. The gauge therefore continues referencing the `LogTablet` created by an earlier test.
`ReplicaManager.shutdown()` does not remove all table and bucket metric groups, so this state survives until another test runs in the reused fork.
### How to reproduce
Use JDK 11 and force the polluting test and the affected tests to run sequentially in one reused fork:
```bash
JAVA_HOME=/path/to/jdk-11 \
./mvnw -pl fluss-server \
-DskipITs \
-Dtest='RemoteLogFetcherTest#testFetchOverlappingSegmentsFromReplicasWithDifferentBoundaries,ReplicaTest#testBucketPhysicalStorageLocalLogSizeIncludesFollower+testPhysicalStorageLocalLogSizeIsScopedPerBucket' \
-Dfluss.forkCount=1 \
-Dfluss.reuseForks=true \
-Dsurefire.runOrder=alphabetical \
test
```
The first test succeeds, while both `ReplicaTest` methods fail with:
```text
expected: 450L
but was: 10830L
```
Running only the two `ReplicaTest` methods succeeds, confirming that the failure is caused by state leaked from the preceding test.
### Solution
Create a fresh `TabletServerMetricGroup` and its associated `UserMetrics` for every `ReplicaTestBase` test instead of using the static instances from `TestingMetricGroups`.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in ReplicaTestBase and inspect how TabletServerMetricGroup and UserMetrics are obtained from TestingMetricGroups. Run the listed Maven command with one reused fork, then verify the two ReplicaTest methods return 450L after each test receives fresh metric groups and that the isolated tests still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100