apache / apache/fluss

[test] ReplicaTest is flaky due to stale gauges in shared test metric groups

Open
#4,020 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

### Problem

`ReplicaTest` may read a stale `physicalStorage/localLogSize` gauge left by another test running in the same reused Maven Surefire JVM.

The following assertions can fail:

```text
ReplicaTest.testBucketPhysicalStorageLocalLogSizeIncludesFollower
expected: 450L
but was: 10830L

ReplicaTest.testPhysicalStorageLocalLogSizeIsScopedPerBucket
expected: 450L
but was: 10830L
```

Running `ReplicaTest` alone, for example directly from IntelliJ IDEA, normally succeeds. The failure only occurs when another test using the same table and bucket registers the metric first in the same reused Surefire fork.

The default test configuration uses two Surefire forks with fork reuse enabled. Therefore, whether the affected tests fail depends on test-class assignment and execution order, making the CI failure flaky.

### Expected behavior

Each test should use metrics associated with the `LogTablet` created by that test. The local log size gauge should return `450L`.

### Actual behavior

The test may reuse a bucket metric group created by an earlier test and read its stale gauge. In the observed failure, the gauge still references a previous `LogTablet` whose size is `10830L`.

### Root cause

`ReplicaTestBase` uses the static singleton instances:

```java
TestingMetricGroups.TABLET_SERVER_METRICS
TestingMetricGroups.USER_METRICS
```

These objects are shared by all tests executed in the same JVM.

`TabletServerMetricGroup` and `TableMetricGroup` cache metric groups using `computeIfAbsent`. When another replica for the same table and bucket registers the same gauge, the existing metric is retained. The gauge therefore continues referencing the `LogTablet` created by an earlier test.

`ReplicaManager.shutdown()` does not remove all table and bucket metric groups, so this state survives until another test runs in the reused fork.

### How to reproduce

Use JDK 11 and force the polluting test and the affected tests to run sequentially in one reused fork:

```bash
JAVA_HOME=/path/to/jdk-11 \
./mvnw -pl fluss-server \
-DskipITs \
-Dtest='RemoteLogFetcherTest#testFetchOverlappingSegmentsFromReplicasWithDifferentBoundaries,ReplicaTest#testBucketPhysicalStorageLocalLogSizeIncludesFollower+testPhysicalStorageLocalLogSizeIsScopedPerBucket' \
-Dfluss.forkCount=1 \
-Dfluss.reuseForks=true \
-Dsurefire.runOrder=alphabetical \
test
```

The first test succeeds, while both `ReplicaTest` methods fail with:

```text
expected: 450L
but was: 10830L
```

Running only the two `ReplicaTest` methods succeeds, confirming that the failure is caused by state leaked from the preceding test.

### Solution

Create a fresh `TabletServerMetricGroup` and its associated `UserMetrics` for every `ReplicaTestBase` test instead of using the static instances from `TestingMetricGroups`.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ReplicaTestBase and inspect how TabletServerMetricGroup and UserMetrics are obtained from TestingMetricGroups. Run the listed Maven command with one reused fork, then verify the two ReplicaTest methods return 450L after each test receives fresh metric groups and that the isolated tests still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.