ClickHouse / ClickHouse/clickhouse-java

Flaky test: R2DBCTestKitImplTest.batch intermittently fails with "Connection pool shut down"

Open
#3,058 0 comments 0 reactions 0 assignees View on GitHub
bug r2dbc test
Dominant language
Java
Stars
1.6k
Forks
636
Avg merge
2d 16h
Merged PRs (30d)
28

Description

## Description

`com.clickhouse.r2dbc.spi.test.R2DBCTestKitImplTest.batch` (module `clickhouse-r2dbc`) fails intermittently in CI. The other 31 tests of the same class pass in the same run, and the sibling matrix legs (other r2dbc-spi versions / server versions) stay green, so an unrelated PR looks like it introduced a regression.

The failure is not a query failure: the Apache HttpClient connection manager is already shut down before the request leases a connection, and the test fails in ~0.04 s.

### Steps to reproduce
1. Run the R2DBC CI job: `mvn --batch-mode --projects clickhouse-r2dbc -DclickhouseVersion= -D'r2dbc-spi.version=' -Dprotocol=http verify`.
2. Repeat. The failure is intermittent — a re-run of the identical commit passes.

### Error Log or Exception StackTrace

```
java.lang.AssertionError: expectation "expectComplete" failed
(expected: onComplete(); actual: onError(java.lang.IllegalStateException: Connection pool shut down))
Suppressed: java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163)
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(...)
...
at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:301)
at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:196)
```

### Evidence of non-determinism

Same commit, same check, fail then pass — run `32022532612` on PR #3056:

* attempt 1, leg `R2DBC 1.0.0.RELEASE + CH 26.3 (http)` → **failure**, `Tests run: 32, Failures: 1`, `batch` 0.041 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/32022532612/job/95368314131
* attempt 2, same leg, same head `7c4b7a66` → **success** (all six R2DBC legs green).

An independent second observation, two weeks earlier and on a different PR and matrix leg — PR #3010, head `d341a338`, which touches only `jdbc-v2` (`SqlParserFacade` + its tests) and cannot reach the R2DBC/HTTP path:

* leg `R2DBC 0.9.1.RELEASE + CH latest (http)` → **failure**, identical signature, `batch` 0.035 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/30856790714/job/91832373067
* the other five R2DBC legs of that same run → success.

### Local reproduction

Does **not** reproduce on an idle or loaded devbox (external ClickHouse 26.3, JDK 17, `-Dprotocol=http`, `r2dbc-spi 1.0.0.RELEASE`):

* 25 × full `R2DBCTestKitImplTest` (32 tests) with 6 busy CPU loops → 0 failures.
* 12 × `R2DBCTestKitImplTest#batch` pinned to a single CPU (`taskset -c 0`) with 8 busy CPU loops → 0 failures.

Total 37 local runs, 0 failures. CI runners are slower and more contended, which is consistent with the CI-only appearance. The job configuration itself notes thread pressure in this environment (`build.yml`: *"http_client and apache_http_client do not work in CI environment (due to limited threads?)"*).

### Expected Behaviour

`batch` completes deterministically. No request should be able to start after the connection that owns its HTTP connection manager has been closed.

### Candidate mechanism (hypothesis — not proven)

`batch` is the only test in the class that runs **two statements through one `Batch`**, and the batch path executes them concurrently:

* `ClickHouseBatch.execute()` (`clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/ClickHouseBatch.java:32-38`) maps the SQL list to `Mono.fromFuture(request::execute)` over a **single shared, mutable** `ClickHouseRequest`, then `flatMap`s them — `flatMap` subscribes inner sources concurrently (default concurrency 256), and the requests carry `ClickHouseClientOption.ASYNC = true` (set in `ClickHouseConnection.createBatch()`), so the actual POST runs on a client executor thread rather than the subscribing thread.
* `ClickHouseConnection.close()` (`clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/connection/ClickHouseConnection.java:70-78`) calls `client.close()`, which closes that connection's `PoolingHttpClientConnectionManager`. `Flux.usingWhen(..., Connection::close)` in `TestKit.batch` triggers it as soon as the outer `Flux` terminates.

If the outer `Flux` can terminate while one of the two batch requests has been submitted to the executor but has not yet leased a connection, the close wins the race and that POST fails at `LaxConnPool.lease` — exactly the observed signature and the sub-100 ms failure time. This would make it a **client-side race in product code, not test debt**: an application closing an R2DBC connection right after a batch completes could see the same `IllegalStateException` instead of a clean completion.

We could not confirm this locally, so it is offered as a starting point only; no root cause is claimed. Note also #2976, a separate confirmed R2DBC connection-lifecycle defect in the same area.

### Suggested direction

Make the batch's completion synchronize with all of its in-flight requests, so the connection cannot be closed while one is still starting (e.g. execute the batch statements sequentially with `concatMap`, or gate the connection close on the completion of every submitted request), instead of retrying or ignoring the failure in the test.

### Configuration

#### Environment
* [ ] Cloud
* Client version: `main` (0.10.0-rc1-SNAPSHOT)
* Language version: JDK 17 (temurin)
* OS: ubuntu-latest (CI); Debian container (local attempt)

#### ClickHouse Server
* ClickHouse Server version: observed on `26.3` and on `latest`; not reproducible locally on 26.3
* Non-default settings: CI test fixtures (`users.d/users.xml` from the repo), `custom_http_params=async_insert=0`
* Tables involved: created by the r2dbc-spi `TestKit` (`CREATE TABLE test ( test_value INTEGER ) ENGINE = Memory`)

---

Found by automated CI monitoring of our own PRs across unrelated runs, then verified by re-checking the run history and attempting a local reproduction; filed as CI-observed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.