apache / apache/pulsar

[Test] Flaky ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover

Open
#26,552 0 comments 0 reactions 1 assignee Claimed by @merlimat View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Describe the bug

`ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover` can assert before a coalesced auto-scale evaluation finishes.

Observed in [CI - Unit - Brokers - Broker Group 2](https://github.com/apache/pulsar/actions/runs/34657945446/job/103455924792), on commit `ba600bde951caa5697f7a8dce91b91586306bb5a`:

```text
java.lang.AssertionError: a hot-segment split must fire — the rollover is not a split expected [2] but found [1]
at ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover(ScalableTopicControllerAutoScaleTest.java:507)
```

The downloaded test report contains two failing attempts. In both, the logs show the expected split dispatched on the scheduler with `trigger=coalesced`, followed by creation of the two successor segments.

### Timing issue

The test registers a consumer, reports hot-segment load, waits on `evaluateAutoScaleForTest().get()`, and immediately asserts the active segment count.

However, [evaluateAndAct](https://github.com/apache/pulsar/blob/ba600bde951caa5697f7a8dce91b91586306bb5a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java#L368-L404) returns an already-completed future when another evaluation is in flight. It marks a pending evaluation, which is later scheduled asynchronously. Consequently, waiting on the test helper does not necessarily wait for the requested evaluation or its split.

This explains the timing-dependent assertion: the test can observe one active segment before the scheduler completes the split. The controller and test are unchanged by the PR where this occurred.

### Suggested correction

Wait for the expected layout after triggering evaluation, or make the test helper await the coalesced evaluation. Preserve the assertions that rebucket cooldown survives failover and does not suppress a load-driven split.

### Environment

- GitHub Actions, Linux, Corretto JDK 25
- Gradle 9.7.1
- [Build scan](https://gradle.com/s/3g5nwuaw7mp3m)

Evidence is from CI logs, the test-report artifact, and source inspection; no local reproduction run yet.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.