ClickHouse / ClickHouse/clickhouse-java
[client-v1] Every compressed read fails on ClickHouse 26.9+: Lz4InputStream hardcodes the LZ4 magic byte but the server default codec is now ZSTD
- Dominant language
- Java
- Stars
- 1.6k
- Forks
- 636
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 28
Description
## Description
On ClickHouse Server **26.9+**, every compressed read through the legacy **v1 client stack**
(`ClickHouseClient` / `clickhouse-http-client`) fails with `Magic is not correct - expect [-126] but got [-112]`.
The v1 HTTP client requests server-side compression with the native `compress=1` framing and then
decompresses it with `com.clickhouse.data.stream.Lz4InputStream`, which rejects any block whose
method byte is not `0x82` (LZ4). ClickHouse 26.9 changed the default codec of that framing to
ZSTD, whose method byte is `0x90`, so the stream now aborts on the very first block.
This affects the v1 client **out of the box**: `ClickHouseClientOption.COMPRESS` defaults to
`true` and `COMPRESS_ALGORITHM` defaults to `LZ4`, so no explicit configuration is needed to hit it.
Disabling compression is the only workaround.
> **Relationship to #3105.** #3105 reports the same class of failure in **client-v2**
> (`client-v2` `ClickHouseLZ4InputStream`, message `Invalid LZ4 magic byte`). This issue is a
> **separate code path** in a different module — `clickhouse-data` `Lz4InputStream`, message
> `Magic is not correct` — which PR #3106 does not touch. The v1 stack stays broken after #3106 merges.
### Steps to reproduce
1. Run a ClickHouse server **26.9+** (verified on `26.9.1.954`).
2. Read any result set with the v1 client using default options (compression on, algorithm LZ4).
3. The read fails on the first block; the same code against a 25.8 server succeeds.
### Error Log or Exception StackTrace
```
FAILED UncheckedIOException: java.io.IOException: Magic is not correct - expect [-126] but got [-112]
root: java.io.IOException: Magic is not correct - expect [-126] but got [-112]
at com.clickhouse.data.stream.Lz4InputStream.updateBuffer(Lz4InputStream.java:64)
```
### Expected Behaviour
The v1 client should decompress a `compress=1` response according to the **method byte the server
actually sent**, rather than assuming LZ4 — i.e. dispatch on the block header and decompress
LZ4 (`0x82`), ZSTD (`0x90`), or none (`0x02`).
The server is the source of truth here. Same query, `compress=1`, two server versions — byte 16 is
the method byte:
```
# ClickHouse 25.8.28.1 -> 0x82 = LZ4
35 ef b4 f9 60 b0 aa a0 61 3a 75 eb 83 88 4a 0c
82 2e 01 00 00 22 01 00
# ClickHouse 26.9.1.954 -> 0x90 = ZSTD (payload begins 28 b5 2f fd = ZSTD frame magic)
26 92 15 1b fd 74 9e e7 06 e7 32 c0 b4 28 31 49
90 9c 00 00 00 22 01 00 00 28 b5 2f fd 60 22 00
```
### Code Example
```java
ClickHouseNode node = ClickHouseNode.builder()
.host("clickhouse").port(ClickHouseProtocol.HTTP, 8123).database("default")
.credentials(ClickHouseCredentials.fromUserAndPassword("default", "***"))
.build();
try (ClickHouseClient client = ClickHouseClient.newInstance(ClickHouseProtocol.HTTP);
ClickHouseResponse response = client.read(node)
.query("SELECT number FROM numbers(1000)")
.option(ClickHouseClientOption.COMPRESS, true) // default is already true
.executeAndWait()) {
for (ClickHouseRecord r : response.records()) {
r.getValue(0).asLong();
}
}
```
Observed with the identical program against two servers:
| Server | compression disabled | compression enabled (default) | explicit LZ4 |
|---|---|---|---|
| 25.8.28.1 | OK | OK | OK |
| 26.9.1.954 | OK | **FAILS** | **FAILS** |
### Root cause
`clickhouse-data/src/main/java/com/clickhouse/data/stream/Lz4InputStream.java:64-67`:
```java
} else if (header[16] != MAGIC) { // MAGIC = (byte) 0x82
throw new IOException(
ClickHouseUtils.format("Magic is not correct - expect [%d] but got [%d]", MAGIC, header[16]));
}
```
The method byte is validated as LZ4 instead of being used to select a decompressor. The stream is
reached from `com.clickhouse.data.compress.Lz4Support.DefaultImpl.decompress(...)`, which is
selected whenever the response compression algorithm is `LZ4` — the default. The corresponding
request in `ClickHouseHttpConnection` is what asks for the native framing:
```java
if (config.isResponseCompressed()) {
if (config.getResponseCompressAlgorithm() == ClickHouseCompression.LZ4) {
appendQueryParameter(builder, "compress", "1");
}
```
### Suggested fix
Dispatch on `header[16]` instead of asserting it, mirroring the approach taken for client-v2 in
PR #3106: accept `0x82` (LZ4), `0x90` (ZSTD), and `0x02` (uncompressed), and raise a clear error
only for a genuinely unknown method byte — which should stay a negative test case.
Two things worth a maintainer decision, which is why this is filed rather than patched:
- **Packaging.** `zstd-jni` is an `optional` dependency of `clickhouse-data`, so adding a ZSTD
decode path there needs a decision about whether it becomes required, or whether the ZSTD branch
fails with an actionable "add zstd-jni" message when the class is absent.
- **Scope.** The v1 stack and `Lz4InputStream` are marked `@Deprecated`. If v1 is not intended to
support 26.9+ servers, an explicit error telling the user to disable compression or migrate would
be preferable to the current low-level magic-byte failure.
### Configuration
#### Client Configuration
```java
// defaults only
ClickHouseClientOption.COMPRESS // true
ClickHouseClientOption.COMPRESS_ALGORITHM // LZ4
```
#### Environment
* [ ] Cloud
* Client version: `main` @ `be331d4ed` (0.10.0-rc1-SNAPSHOT)
* Language version: OpenJDK 17.0.20
* OS: Linux (x86_64, Docker)
#### ClickHouse Server
* ClickHouse Server version: **26.9.1.954** (fails) / 25.8.28.1 (works)
* ClickHouse Server non-default settings, if any: none relevant to the framing
* `CREATE TABLE` statements for tables involved: none — reproduces with `SELECT number FROM numbers(1000)`
* Sample data for all these tables: n/a
---
*Found by automated analysis while working on #3105, and verified against live 26.9 and 25.8 servers
rather than by inspection. The v1 JDBC driver (`clickhouse-jdbc`) was checked separately and fails
with the client-v2 message instead, so that surface is covered by #3105, not by this issue.*
Contributor guide
Assessment
This issue has not been assessed yet.