ClickHouse / ClickHouse/clickhouse-java
[client-v2] Every compressed read fails on ClickHouse 26.9+: response reader is hardcoded to LZ4 but the server default codec is now ZSTD(3)
- Ngôn ngữ chính
- Java
- Star
- 1.6k
- Fork
- 636
- Merge trung bình
- 2 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 29
Mô tả
### Summary
ClickHouse [#108786](https://github.com/ClickHouse/ClickHouse/pull/108786) ("Switch the default compression to ZSTD(3) for table data and network", merged 2026-09-05, `Version info` says merged into `26.9.1.810`, so included in 26.9 and later) changes `CompressionCodecFactory::getDefaultCodec` from `LZ4` to `ZSTD(3)`.
The HTTP `compress=1` framed output is one of the direct `getDefaultCodec` users. client-v2 requests `compress=1` by default and pipes the response body through `ClickHouseLZ4InputStream`, which asserts the LZ4 block magic byte, so **every compressed read fails against ClickHouse 26.9+**:
```
com.clickhouse.client.api.ClientException: Invalid LZ4 magic byte: '-112'
at com.clickhouse.client.api.internal.ClickHouseLZ4InputStream.refill(ClickHouseLZ4InputStream.java:112)
at com.clickhouse.client.api.internal.ClickHouseLZ4InputStream.read(ClickHouseLZ4InputStream.java:62)
...
at com.clickhouse.client.api.internal.TableSchemaParser.readTSKV(TableSchemaParser.java:23)
at com.clickhouse.client.api.Client.getTableSchemaImpl(Client.java:1872)
at com.clickhouse.client.api.Client.getTableSchema(Client.java:1847)
```
`getTableSchema` is just where we hit it first — this is not specific to schema introspection, it affects any compressed response.
Per that PR's own changelog, the HTTP `compress=1` path has **no runtime rollback**: it is not controlled by the `compatibility` setting, per-column `CODEC`, or the server `` config. So this cannot be worked around server-side.
### Affected versions
- client-v2 **0.9.5** — verified failing
- client-v2 **0.10.0** (latest release) — same `ClickHouseLZ4InputStream` with the same `Invalid LZ4 magic byte` check, so also affected
- Server: ClickHouse **26.9+** (reproduced on `clickhouse/clickhouse-server:head` = `26.9.1.943`). 26.8 and earlier are unaffected by *this* bug.
### Root cause / evidence
`-112` is `0x90` signed. Raw `curl` against 26.9.1.943 confirms the framing:
```
$ curl -s -u default:x 'localhost:8123/?compress=1' --data-binary 'DESCRIBE TABLE t FORMAT TSKV' | xxd | head -2
00000000: 151e 70e4 4490 8555 b0c4 8424 d0d8 731d ..p.D..U...$..s.
00000010: 9065 0000 00c0 0000 0028 b52f fd20 c09d .e.......(./. ..
```
After the 16-byte checksum, offset `0x10` is `0x90` — the compression method byte for ZSTD, exactly the byte reported in the exception — and offset `0x18` is `28 b5 2f fd`, the ZSTD magic number. `ClickHouseLZ4InputStream` expects `0x82` there.
The same request without `compress=1` returns plain readable TSKV.
### Reproduce
```bash
docker run -d --name chhead -e CLICKHOUSE_PASSWORD=x -p 8123:8123 clickhouse/clickhouse-server:head
# then any client-v2 read, e.g.
# client.getTableSchema("t", "default")
```
### Suggested fix
The compressed frame is self-describing — ClickHouse#108786 explicitly relies on "the receiver auto-detects the codec", which holds for the server's own readers. client-v2 does not auto-detect; it assumes LZ4. The response reader should dispatch on the compression method byte in the frame header (`0x82` LZ4, `0x90` ZSTD, `0x02` none) rather than asserting LZ4, and gain a ZSTD decompression path.
### Note on interaction with #3070 / #3094
On 26.9+ this bug currently *masks* the `X-ClickHouse-Format` precedence bug (#3070, #3094): the stream dies in `refill` before `readTSKV` ever sees bytes. I verified the header bug is still present on 26.9.1.943 — `curl -H 'format: TSKV' ... --data-binary 'DESCRIBE TABLE t'` returns TabSeparated there too. So fixing only this issue will move `getTableSchema` back to failing with `Non-null columnName and columnType are required`; both need to land.
Setting `compressServerResponse(false)` sidesteps this issue alone, but for the same reason it is not a usable workaround for `getTableSchema`, and it costs compression on every read.
### Context
Found while running the [ClickHouse Flink connector](https://github.com/ClickHouse/flink-connector-clickhouse) test suite against ClickHouse HEAD.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với ClickHouseLZ4InputStream.java của client-v2, đặc biệt là đường dẫn refill ở dòng 112, và theo dõi cách các HTTP response đã nén đến được các response reader được getTableSchema và các thao tác đọc khác sử dụng. Tái hiện với clickhouse/clickhouse-server:head cùng compress=1, sau đó xác minh rằng các response LZ4, ZSTD đã nén và các response không nén được xử lý theo các byte phương thức frame tương ứng mà không làm hỏng các thao tác đọc hiện có.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- java
- Lĩnh vực
- databases
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 64/100