Flaky test Triage - 2026-09-18

Start with IngestionBackwardCompatibilityDockerTest and the failing embedded-tests shard, then inspect .github/workflows/cron-job-its.yml and the dependency-check configuration. Compare the reported failures with fixes referenced in #20291 and #20236; the work is done when the affected test shard and security workflow complete without these failures.

This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.

This triage covers the 4 commits merged to master on 2026-09-17. One commit (2fed7ca11d, #20357, the last of the day) had failed jobs, two in total: the push-triggered I*,L*,J* unit-test shard and the security vulnerabilities job of the scheduled Cron Job ITs run. Neither failure is caused by the commit, which is a one-line maven-deploy-plugin version bump in benchmarks/pom.xml. The failing embedded test is a task-slot starvation race between two test methods (the MSQ worker task started and finished successfully 8 ms after a leftover Kafka task was killed, but that happened 8 s after the 60 s timeout); the same shard passed on the PR's own pre-merge run (6b038a68) and on the two master commits merged 1 and 3 minutes earlier (95f3747a08, 0020a4c564), and a fix is already open in #20291. The security job hit a Sonatype OSS Index HTTP 402 outage that affected every dependency-check run on 2026-09-17, on top of the known elasticache-java-cluster-client false positive that has failed every scheduled run since at least 2026-09-14 (fix in #20236). No re-runs had been triggered; embedded-tests run with surefire.rerunFailingTestsCount=0, so the test failed on its single attempt.

Summary

Analysis and suggested fixes

1. IngestionBackwardCompatibilityDockerTest.test_ingestWikipedia1DayWithMSQ_andQueryData (embedded-tests, I*,L*,J* shard)

IngestionBackwardCompatibilityDockerTest extends IngestionSmokeTest and runs its test methods against an Overlord and a Coordinator running in apache/druid:31.0.2 containers, with an embedded Indexer that has druid.worker.capacity=2. In this run JUnit executed test_runKafkaSupervisor first. That test starts a Kafka supervisor with taskDuration=PT0.5S and maxRowsPerSegment=1, waits for the data to become queryable and then only suspends the supervisor. Because of the 500 ms task duration the supervisor had already rolled over to a second task (index_kafka_..._flmlcdof, started at 02:07:46.013) when the suspend arrived, and the Druid 31 Overlord did not shut that task down until 02:08:55, about 68 s later. The MSQ test started at 02:07:47: its controller task query-eae9262c-... took the second (last) slot at 02:07:47.554 and then waited for its single worker task, which the Overlord could not assign anywhere. The 60 s LatchableEmitter timeout expired at about 02:08:47; the Kafka task was killed at 02:08:55.183, the worker task started 8 ms later, and the whole ingestion (24,433 rows, segment pushed to MinIO at 02:08:57.5) completed in about 3 s. So the MSQ job itself is fine; the failure is slot starvation caused by the previous test's leftover task. The non-docker IngestionSmokeTest in the same shard passed (6 tests in 47.8 s) because the current Overlord stops a suspended supervisor's tasks promptly, which is why the race only bites the backward-compatibility variant.

The commit (#20357) only changes the maven-deploy-plugin version in benchmarks/pom.xml and does not touch embedded-tests, MSQ or the Kafka indexing service. The same shard passed on the PR's own pre-merge run and on the master commits merged immediately before (95f3747a08 and 0020a4c564), and the I*,L*,J* shard has not failed on any other master push in the last week. The related LatchableEmitter timeouts in IngestionSmokeTest on JDK 25 are tracked in #19463.

Suggested fix: merge #20291, whose @AfterEach terminates the test's supervisor, cancels every running, pending or waiting task of the test datasource and polls until none is left before marking segments unused, which guarantees that both Indexer slots are free when the next method starts. Independently, test_runKafkaSupervisor could terminate the supervisor instead of suspending it and wait for the task/run/time events of its tasks, and the docker variant could set druid.worker.capacity=3 so a controller plus worker never depends on a slot held by a previous test.

2. security vulnerabilities (cron): Sonatype OSS Index HTTP 402, and false-positive memcached CVEs on elasticache-java-cluster-client-1.2.4.jar

The job runs mvn dependency-check:purge dependency-check:check, so it first re-downloaded the entire NVD feed (393,890 records), which took 5 h 07 min. The first Maven invocation then failed on druid-processing because the OSS Index analyzer received 402 Payment Required from api.guide.sonatype.com ("credits insufficient / payment required, disabling the analyzer"); dependency-check 13 treats an analyzer exception as fatal (failOnError=true). The same 402 appeared in all four security vulnerabilities jobs that ran on 2026-09-17, including the ones for the 38.0.0 release branch, and not in the 2026-09-16 scheduled run, so it is a quota problem on the OSS Index account, unrelated to the commit. A second, unintended scan then produced the druid-server report: the fallback in .github/workflows/cron-job-its.yml embeds `mvn dependency-check:check` in unescaped backticks inside the echo string, so bash executes another full dependency-check:check (without the OSS Index and NVD credentials) as a command substitution before printing the message. That second scan is where elasticache-java-cluster-client-1.2.4.jar was matched to cpe:2.3:a:memcached:memcached:1.2.4 (the C memcached server daemon) purely on the coinciding version string and flagged with 10 server-side CVEs. Druid only uses this jar as a client library in the memcached cache; this is the same false positive triaged in #20360 and #20366, it has failed every scheduled run since 2026-09-14, and there is still no suppression entry for it on master (the 38.0.0 branch got one via #20335).

Suggested fix: merge #20236, which adds a <suppress> entry matching ^pkg:maven/com\.amazonaws/elasticache-java-cluster-client@.*$ against cpe:/a:memcached:memcached, or split that single block into a standalone PR so the cron job can go green. For the OSS Index outage, replenish or rotate the OSS_INDEX_USERNAME/OSS_INDEX_PASSWORD credentials and pass -DossIndexWarnOnlyOnRemoteErrors=true so a remote OSS Index error degrades to a warning instead of failing the build. In the workflow, quote the fallback message with single quotes (or escape the backticks) so it stops launching a second scan. Finally, #20126 (stop purging the NVD database and cache it between runs) would cut the 5 h download and make this job usable again.