Flaky test triage - 2026-09-10

Triage of CI failures on the last 20 `master` commits as of 2026-09-10. 7 commits had at least one failed job (rows for 5e71056845 and 1f68c5e168 added after their runs completed). All failures are flaky tests or CI infrastructure outages, none is caused by the commit itself: every failing test passed on the neighbouring master commits (or on the PR's own pre-merge run of the same shard), and none of the failing runs had been retried.

### Summary

| Commit | Failed job | Failure log | Root cause | Verdict |
|---|---|---|---|---|
| 5e71056845 (#20268, response identity headers) | unit tests (25, C*) | [job 102764139085](https://github.com/apache/druid/actions/runs/34443807042/job/102764139085) | `CompactionTaskTest.testCompactionWithTimestampDimension` (embedded): overlord reported 2 visible segments but `SELECT COUNT(*) FROM sys.segments ...` on the Broker returned an empty result | Flaky |
| 5e71056845 (#20268) | unit tests (25, T*,F*,G*,J*) | [job 102764138933](https://github.com/apache/druid/actions/runs/34443807042/job/102764138933) | `TaskQueueScaleTest.doMassLaunchAndExit`: `running+pending+waiting` count 983/993/997/999 vs expected 1000, failed all 4 attempts | Flaky (race, fix in #20291) |
| 1f68c5e168 (#19818, harden arithmetic/bounds checks) | unit tests (25, S*) | [job 102763954928](https://github.com/apache/druid/actions/runs/34443744711/job/102763954928) | `SeekableStreamSupervisorStateTest.testSupervisorStopTaskGroupEarly`: EasyMock expected `TaskQueue.shutdown("id1", ...)` once, observed 0 times, failed all 4 attempts | Flaky (race) |
| 8d9545ea58 (#20247, clone sync criteria) | docker-tests | [job 102752252468](https://github.com/apache/druid/actions/runs/34439796339/job/102752252468) | `IngestionDockerTest.test_runIndexTask_andKillData` timed out after 240s waiting for `segment/schemaCache/dataSource/removed` from the broker | Flaky |
| 61ed0a389d (#20272, portable printf) | unit tests (25, R*,B*,Q*,V*) | [job 102731457738](https://github.com/apache/druid/actions/runs/34432737655/job/102731457738) | `ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM` expected leak count 1, got 3; `QueryLaningTest.test_queryUsesLaneInQueryContext_inManualStrategy` timed out after 100s waiting for segments to become available | Flaky |
| 61ed0a389d (#20272) | docker-tests | [job 102731457503](https://github.com/apache/druid/actions/runs/34432737655/job/102731457503) | `KubernetesClusterDockerTest` setup: k3s pod `druid-router-*` never became Ready | Flaky / infra |
| 8ff36f7ef8 (#20292, granularity lookups) | security vulnerabilities (cron) | [job 102731264787](https://github.com/apache/druid/actions/runs/34432672790/job/102731264787) | OWASP dependency-check: `NvdApiException: NVD Returned Status Code: 503` | Infra |
| 3985d927a6 (#20271, compact object headers) | unit tests (25, K*,E*,W*,Z*,Y*,X*) | [job 102727804299](https://github.com/apache/druid/actions/runs/34431507380/job/102727804299) | `KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored`: EasyMock unexpected call `peonClient.deleteCompletedPeonJobsOlderThan(...)`, failed all 4 attempts | Flaky (race) |
| c7c73625a5 (#20277, jetty bump) | unit tests (25, S*) | [job 102713226586](https://github.com/apache/druid/actions/runs/34426655504/job/102713226586) | `setup_test_profiling_env.sh` exited 35 before Maven started: curl SSL connect error downloading `jfr-profiler-1.0.0.jar` from static.imply.io | Infra |

### Analysis and suggested fixes

**1. `KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored`**
`KubernetesTaskRunner.start()` schedules `client.deleteCompletedPeonJobsOlderThan(...)` on a real `ScheduledExecutorService` with a 1 ms initial delay. Whether it fires before the test's `verifyAll()` depends on thread scheduling, so the strict `@Mock peonClient` sometimes sees an unexpected call. It failed 4/4 attempts in one run because the retries run in the same loaded JVM. The commit only added a JVM flag to surefire and is unrelated. #20285 hit the same failure independently.
Suggested fix: expect the cleanup call with `.anyTimes()` (or use a nice mock for `peonClient`); better, inject the cleanup `ScheduledExecutorService` into `KubernetesTaskRunner` so tests can supply a manual/no-op executor.

**2. `ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM`**
`ReferenceCountingResourceHolder.LEAKED_RESOURCES` is a JVM-global static counter and surefire runs with `reuseForks=true`. The test's `System.gc()` loop also collects unclosed holders leaked by earlier test classes in the same fork, so the delta is >1.
Suggested fix: assert `leakedResources() >= initial + 1` together with `released.get()`, or drop the counter assertion and rely only on the `released` flag which is specific to this test.

**3. `QueryLaningTest` / `IngestionDockerTest.test_runIndexTask_andKillData`**
Both wait on a single emitted metric with a fixed timeout in an embedded cluster on a shared GitHub runner; a slow segment load or a missed emission tick trips it.
Suggested fix: poll the actual state (segment availability via `sys.segments`, or datasource absence in the broker schema) with retries instead of waiting for one metric event, as #19416 already did for the `sys.segments` part of this test. Raising the 100s timeout in `EmbeddedClusterApis.waitForAllSegmentsToBeAvailable` is a cheaper stopgap.

**4. `KubernetesClusterDockerTest` (router pod not Ready)**
Same image build and test passed on the very next master commit; the `printf` change in the failing commit is behaviour-preserving.
Suggested fix: on timeout in `K3sClusterResource.waitUntilPodIsReady`, include `kubectl describe pod` output and container logs in the failure message so the next occurrence is diagnosable; consider a longer readiness timeout for the router, which starts last.

**5. Security vulnerabilities cron job (NVD 503)**
Suggested fix: cache the NVD database across runs and/or configure `nvdApiKey` + `nvdMaxRetryCount` in `dependency-check-maven`; treat `UpdateException` as a soft failure.

**6. `setup_test_profiling_env.sh` exit 35**
An external CDN hiccup failed a whole unit test shard before any test ran.
Suggested fix: make the profiler download non-fatal, e.g. `curl --retry 3 ... || { echo "JFR_PROFILER_ARG_LINE="; exit 0; }`.

**7. `CompactionTaskTest.testCompactionWithTimestampDimension` (embedded-tests, C\* shard)**
`CompactionTestBase.verifySegmentsCount` waits for a single `segment/metadataCache/sync/time` event on the Broker and then asserts against `sys.segments`. The first assertion (segment count from the Overlord) passed with 2; only the Broker-side `sys.segments` query returned nothing, i.e. the sync event that was observed predates the compaction result, or the datasource was not yet in the Broker schema. The commit only adds response headers in Jetty and its pre-merge run of the same C\* shard passed.
Suggested fix: in `verifyNumVisibleSegmentsIs` / `verifySegmentsCount`, retry the `sys.segments` query until it matches (with a bounded timeout) instead of asserting once after a single sync event.

**8. `TaskQueueScaleTest.doMassLaunchAndExit` (indexing-service, T\* shard)**
The assertion `numTasks == running + pending + waiting` reads three independent snapshots while tasks are already completing; the test itself carries the comment "in theory we can get a race here". Observed 983, 993, 997, 999 across the 4 attempts, which is consistent with tasks finishing between the snapshots. The same failure was observed on Dependabot PR #20281. #20291 already removes this assertion and waits for cleanup instead of a fixed sleep; merging it should close this one.

**9. `SeekableStreamSupervisorStateTest.testSupervisorStopTaskGroupEarly` (indexing-service, S\* shard)**
The test drains the supervisor's notice queue with `while (getNoticesQueueSize() > 0) sleep(100)` and then calls `runInternal()` and `verifyAll()`. The queue size drops to 0 when a notice is *taken*, not when it has finished executing, so `runInternal()` can run before the handoff notice has marked the task group for early stop, and `TaskQueue.shutdown(...)` is never called. Failed 4/4 attempts under load; the same failure hit PR #20277 on an unrelated jetty bump. The commit only touches test files and does not touch this test or the supervisor.
Suggested fix: wait for a completion signal rather than queue emptiness, e.g. expose a "notices processed" counter or latch in the test supervisor, or have `handoffTaskGroupsEarly` return a future the test can join. Alternatively run the handoff notice synchronously in the test subclass.

Choose one failure first, then read the named test and its referenced entry point, such as KubernetesTaskRunnerTest, ReferenceCountingResourceHolderTest, CompactionTestBase.verifySegmentsCount, or setup_test_profiling_env.sh. Run the focused test or CI job, confirm the race or infrastructure failure, and finish with a bounded fix plus a passing regression run.

Written by the indexing model from the issue text.