ClickHouse / ClickHouse/clickhouse-java
Flaky test: R2DBCTestKitImplTest.batch intermittently fails with "Connection pool shut down"
- Dominant language
- Java
- Stars
- 1.6k
- Forks
- 636
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 28
Description
## Description
`com.clickhouse.r2dbc.spi.test.R2DBCTestKitImplTest.batch` (module `clickhouse-r2dbc`) fails intermittently in CI. The other 31 tests of the same class pass in the same run, and the sibling matrix legs (other r2dbc-spi versions / server versions) stay green, so an unrelated PR looks like it introduced a regression.
The failure is not a query failure: the Apache HttpClient connection manager is already shut down before the request leases a connection, and the test fails in ~0.04 s.
### Steps to reproduce
1. Run the R2DBC CI job: `mvn --batch-mode --projects clickhouse-r2dbc -DclickhouseVersion= -D'r2dbc-spi.version=' -Dprotocol=http verify`.
2. Repeat. The failure is intermittent — a re-run of the identical commit passes.
### Error Log or Exception StackTrace
```
java.lang.AssertionError: expectation "expectComplete" failed
(expected: onComplete(); actual: onError(java.lang.IllegalStateException: Connection pool shut down))
Suppressed: java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163)
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(...)
...
at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:301)
at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:196)
```
### Evidence of non-determinism
Same commit, same check, fail then pass — run `32022532612` on PR #3056:
* attempt 1, leg `R2DBC 1.0.0.RELEASE + CH 26.3 (http)` → **failure**, `Tests run: 32, Failures: 1`, `batch` 0.041 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/32022532612/job/95368314131
* attempt 2, same leg, same head `7c4b7a66` → **success** (all six R2DBC legs green).
An independent second observation, two weeks earlier and on a different PR and matrix leg — PR #3010, head `d341a338`, which touches only `jdbc-v2` (`SqlParserFacade` + its tests) and cannot reach the R2DBC/HTTP path:
* leg `R2DBC 0.9.1.RELEASE + CH latest (http)` → **failure**, identical signature, `batch` 0.035 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/30856790714/job/91832373067
* the other five R2DBC legs of that same run → success.
### Local reproduction
Does **not** reproduce on an idle or loaded devbox (external ClickHouse 26.3, JDK 17, `-Dprotocol=http`, `r2dbc-spi 1.0.0.RELEASE`):
* 25 × full `R2DBCTestKitImplTest` (32 tests) with 6 busy CPU loops → 0 failures.
* 12 × `R2DBCTestKitImplTest#batch` pinned to a single CPU (`taskset -c 0`) with 8 busy CPU loops → 0 failures.
Total 37 local runs, 0 failures. CI runners are slower and more contended, which is consistent with the CI-only appearance. The job configuration itself notes thread pressure in this environment (`build.yml`: *"http_client and apache_http_client do not work in CI environment (due to limited threads?)"*).
### Expected Behaviour
`batch` completes deterministically. No request should be able to start after the connection that owns its HTTP connection manager has been closed.
### Candidate mechanism (hypothesis — not proven)
`batch` is the only test in the class that runs **two statements through one `Batch`**, and the batch path executes them concurrently:
* `ClickHouseBatch.execute()` (`clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/ClickHouseBatch.java:32-38`) maps the SQL list to `Mono.fromFuture(request::execute)` over a **single shared, mutable** `ClickHouseRequest`, then `flatMap`s them — `flatMap` subscribes inner sources concurrently (default concurrency 256), and the requests carry `ClickHouseClientOption.ASYNC = true` (set in `ClickHouseConnection.createBatch()`), so the actual POST runs on a client executor thread rather than the subscribing thread.
* `ClickHouseConnection.close()` (`clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/connection/ClickHouseConnection.java:70-78`) calls `client.close()`, which closes that connection's `PoolingHttpClientConnectionManager`. `Flux.usingWhen(..., Connection::close)` in `TestKit.batch` triggers it as soon as the outer `Flux` terminates.
If the outer `Flux` can terminate while one of the two batch requests has been submitted to the executor but has not yet leased a connection, the close wins the race and that POST fails at `LaxConnPool.lease` — exactly the observed signature and the sub-100 ms failure time. This would make it a **client-side race in product code, not test debt**: an application closing an R2DBC connection right after a batch completes could see the same `IllegalStateException` instead of a clean completion.
We could not confirm this locally, so it is offered as a starting point only; no root cause is claimed. Note also #2976, a separate confirmed R2DBC connection-lifecycle defect in the same area.
### Suggested direction
Make the batch's completion synchronize with all of its in-flight requests, so the connection cannot be closed while one is still starting (e.g. execute the batch statements sequentially with `concatMap`, or gate the connection close on the completion of every submitted request), instead of retrying or ignoring the failure in the test.
### Configuration
#### Environment
* [ ] Cloud
* Client version: `main` (0.10.0-rc1-SNAPSHOT)
* Language version: JDK 17 (temurin)
* OS: ubuntu-latest (CI); Debian container (local attempt)
#### ClickHouse Server
* ClickHouse Server version: observed on `26.3` and on `latest`; not reproducible locally on 26.3
* Non-default settings: CI test fixtures (`users.d/users.xml` from the repo), `custom_http_params=async_insert=0`
* Tables involved: created by the r2dbc-spi `TestKit` (`CREATE TABLE test ( test_value INTEGER ) ENGINE = Memory`)
---
Found by automated CI monitoring of our own PRs across unrelated runs, then verified by re-checking the run history and attempting a local reproduction; filed as CI-observed.
Contributor guide
Assessment
This issue has not been assessed yet.