ClickHouse / ClickHouse/clickhouse-java
[client-v1] Every compressed read fails on ClickHouse 26.9+: Lz4InputStream hardcodes the LZ4 magic byte but the server default codec is now ZSTD
- 主要言語
- Java
- スター
- 1.6k
- フォーク
- 636
- 平均マージ
- 2日 23時間
- マージ済み PR(30日)
- 29
説明
## Description
On ClickHouse Server **26.9+**, every compressed read through the legacy **v1 client stack**
(`ClickHouseClient` / `clickhouse-http-client`) fails with `Magic is not correct - expect [-126] but got [-112]`.
The v1 HTTP client requests server-side compression with the native `compress=1` framing and then
decompresses it with `com.clickhouse.data.stream.Lz4InputStream`, which rejects any block whose
method byte is not `0x82` (LZ4). ClickHouse 26.9 changed the default codec of that framing to
ZSTD, whose method byte is `0x90`, so the stream now aborts on the very first block.
This affects the v1 client **out of the box**: `ClickHouseClientOption.COMPRESS` defaults to
`true` and `COMPRESS_ALGORITHM` defaults to `LZ4`, so no explicit configuration is needed to hit it.
Disabling compression is the only workaround.
> **Relationship to #3105.** #3105 reports the same class of failure in **client-v2**
> (`client-v2` `ClickHouseLZ4InputStream`, message `Invalid LZ4 magic byte`). This issue is a
> **separate code path** in a different module — `clickhouse-data` `Lz4InputStream`, message
> `Magic is not correct` — which PR #3106 does not touch. The v1 stack stays broken after #3106 merges.
### Steps to reproduce
1. Run a ClickHouse server **26.9+** (verified on `26.9.1.954`).
2. Read any result set with the v1 client using default options (compression on, algorithm LZ4).
3. The read fails on the first block; the same code against a 25.8 server succeeds.
### Error Log or Exception StackTrace
```
FAILED UncheckedIOException: java.io.IOException: Magic is not correct - expect [-126] but got [-112]
root: java.io.IOException: Magic is not correct - expect [-126] but got [-112]
at com.clickhouse.data.stream.Lz4InputStream.updateBuffer(Lz4InputStream.java:64)
```
### Expected Behaviour
The v1 client should decompress a `compress=1` response according to the **method byte the server
actually sent**, rather than assuming LZ4 — i.e. dispatch on the block header and decompress
LZ4 (`0x82`), ZSTD (`0x90`), or none (`0x02`).
The server is the source of truth here. Same query, `compress=1`, two server versions — byte 16 is
the method byte:
```
# ClickHouse 25.8.28.1 -> 0x82 = LZ4
35 ef b4 f9 60 b0 aa a0 61 3a 75 eb 83 88 4a 0c
82 2e 01 00 00 22 01 00
# ClickHouse 26.9.1.954 -> 0x90 = ZSTD (payload begins 28 b5 2f fd = ZSTD frame magic)
26 92 15 1b fd 74 9e e7 06 e7 32 c0 b4 28 31 49
90 9c 00 00 00 22 01 00 00 28 b5 2f fd 60 22 00
```
### Code Example
```java
ClickHouseNode node = ClickHouseNode.builder()
.host("clickhouse").port(ClickHouseProtocol.HTTP, 8123).database("default")
.credentials(ClickHouseCredentials.fromUserAndPassword("default", "***"))
.build();
try (ClickHouseClient client = ClickHouseClient.newInstance(ClickHouseProtocol.HTTP);
ClickHouseResponse response = client.read(node)
.query("SELECT number FROM numbers(1000)")
.option(ClickHouseClientOption.COMPRESS, true) // default is already true
.executeAndWait()) {
for (ClickHouseRecord r : response.records()) {
r.getValue(0).asLong();
}
}
```
Observed with the identical program against two servers:
| Server | compression disabled | compression enabled (default) | explicit LZ4 |
|---|---|---|---|
| 25.8.28.1 | OK | OK | OK |
| 26.9.1.954 | OK | **FAILS** | **FAILS** |
### Root cause
`clickhouse-data/src/main/java/com/clickhouse/data/stream/Lz4InputStream.java:64-67`:
```java
} else if (header[16] != MAGIC) { // MAGIC = (byte) 0x82
throw new IOException(
ClickHouseUtils.format("Magic is not correct - expect [%d] but got [%d]", MAGIC, header[16]));
}
```
The method byte is validated as LZ4 instead of being used to select a decompressor. The stream is
reached from `com.clickhouse.data.compress.Lz4Support.DefaultImpl.decompress(...)`, which is
selected whenever the response compression algorithm is `LZ4` — the default. The corresponding
request in `ClickHouseHttpConnection` is what asks for the native framing:
```java
if (config.isResponseCompressed()) {
if (config.getResponseCompressAlgorithm() == ClickHouseCompression.LZ4) {
appendQueryParameter(builder, "compress", "1");
}
```
### Suggested fix
Dispatch on `header[16]` instead of asserting it, mirroring the approach taken for client-v2 in
PR #3106: accept `0x82` (LZ4), `0x90` (ZSTD), and `0x02` (uncompressed), and raise a clear error
only for a genuinely unknown method byte — which should stay a negative test case.
Two things worth a maintainer decision, which is why this is filed rather than patched:
- **Packaging.** `zstd-jni` is an `optional` dependency of `clickhouse-data`, so adding a ZSTD
decode path there needs a decision about whether it becomes required, or whether the ZSTD branch
fails with an actionable "add zstd-jni" message when the class is absent.
- **Scope.** The v1 stack and `Lz4InputStream` are marked `@Deprecated`. If v1 is not intended to
support 26.9+ servers, an explicit error telling the user to disable compression or migrate would
be preferable to the current low-level magic-byte failure.
### Configuration
#### Client Configuration
```java
// defaults only
ClickHouseClientOption.COMPRESS // true
ClickHouseClientOption.COMPRESS_ALGORITHM // LZ4
```
#### Environment
* [ ] Cloud
* Client version: `main` @ `be331d4ed` (0.10.0-rc1-SNAPSHOT)
* Language version: OpenJDK 17.0.20
* OS: Linux (x86_64, Docker)
#### ClickHouse Server
* ClickHouse Server version: **26.9.1.954** (fails) / 25.8.28.1 (works)
* ClickHouse Server non-default settings, if any: none relevant to the framing
* `CREATE TABLE` statements for tables involved: none — reproduces with `SELECT number FROM numbers(1000)`
* Sample data for all these tables: n/a
---
*Found by automated analysis while working on #3105, and verified against live 26.9 and 25.8 servers
rather than by inspection. The v1 JDBC driver (`clickhouse-jdbc`) was checked separately and fails
with the client-v2 message instead, so that surface is covered by #3105, not by this issue.*
コントリビューションガイド
調査の方向性
clickhouse-data/src/main/java/com/clickhouse/data/stream/Lz4InputStream.java から始め、v1 レスポンスパスから Lz4Support.DefaultImpl.decompress がどのように選択されるかを追跡します。framing リクエストについて ClickHouseHttpConnection を確認します。提供された SELECT を使って ClickHouse 25.8 および 26.9 に対して再現し、その後 0x82、0x90、0x02 が適切に処理され、未知のメソッドが明確な失敗を生成することを確認しながら、zstd-jni のパッケージング方針を解決します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- api, backend
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 48/100