ClickHouse / ClickHouse/clickhouse-java

[client-v2, jdbc-v2] A caller-supplied FORMAT clause is silently ignored on ClickHouse 26.8+ (X-ClickHouse-Format wins)

Open
#3,070 0 comments 0 reactions 0 assignees View on GitHub
area:format bug client-api-v2 jdbc-v2
Dominant language
Java
Stars
1.6k
Forks
636
Avg merge
2d 16h
Merged PRs (30d)
28

Description

## Description

On ClickHouse 26.8+ the `X-ClickHouse-Format` request header is an explicit override that
wins over the query's `FORMAT` clause. client-v2 sends that header on **every** request, and
defaults it to `RowBinaryWithNamesAndTypes` when the caller sets no format. A caller who
writes `SELECT ... FORMAT JSONEachRow` therefore receives `RowBinaryWithNamesAndTypes` bytes
instead. There is no exception — only different bytes.

This contradicts the documented contract of `Client.query(...)`:

> Server response format can be specified through `settings` or in SQL query.
> If specified in both, the `sqlQuery` will take precedence.
>
> — `client-v2/src/main/java/com/clickhouse/client/api/Client.java` (Javadoc of `query`)

Related but different: issue #3068 covers the client's own **internal** metadata queries
(`getTableSchema`, `getTableSchemaFromQuery`, `ping`), which fail loudly. This report is the
**caller-facing** half, which fails silently. The two do not share a fix: #3068 is fixed by
changing the client's own SQL, this one is about what the client does with SQL the user wrote.

Server-side cause is the same: ClickHouse PR ClickHouse/ClickHouse#105249 turned
`X-ClickHouse-Format` from an alias of `default_format` (a fallback) into an alias of `format`
(an explicit override).

### Steps to reproduce

1. Start `clickhouse/clickhouse-server:head` (26.8.1.1761).
2. Run `client.query("SELECT 1 AS a, 'x' AS b FORMAT JSONEachRow", new QuerySettings())`.
3. Read `response.getFormat()` and the response body.

Three tests already in the repository pin this contract and turn red on 26.8, while passing
on 26.6.2.160 (same worktree, same build, only the server image changed):

| Test | Result on 26.8.1.1761 |
|---|---|
| `client-v2` `QueryTests.testSettingsNotChanged` | `expected [JSONEachRow] but found [RowBinaryWithNamesAndTypes]` |
| `jdbc-v2` `StatementTest.testJSONEachRowFormatRequiresParserFactory` | `Expected SQLException` (none thrown) |
| `jdbc-v2` `StatementTest.testTextFormatInResponse` | `Expected SQLException to be thrown, but nothing was thrown` |

`jdbc-v2` `StatementTest.testJSONEachRowFormat` still passes, but for the wrong reason: its
assertions (`getInt`, `getString`) hold for either format, so the silent switch to RowBinary
is invisible to it.

The two `jdbc-v2` failures show that the shipped, documented `FORMAT JSONEachRow` support in
the JDBC driver (`jdbc_json_parser_factory`, `docs/client-v2-json-support.md`) is inert on
26.8: `response.getFormat()` can no longer be `JSONEachRow`, so `StatementImpl` never selects
`JSONEachRowFormatReader` and returns binary-decoded rows instead.

### Error Log or Exception StackTrace

```
No exception is thrown. That is the problem.

client-v2 QueryTests.testSettingsNotChanged:2345
expected [JSONEachRow] but found [RowBinaryWithNamesAndTypes]
```

### Expected Behaviour

Either of the following, but not the current silent mismatch:

* the `FORMAT` clause keeps taking precedence, as `Client.query`'s Javadoc states; or
* the `FORMAT` clause is no longer supported and the client raises a clear error, with the
Javadoc, `docs/client-v2-json-support.md`, `docs/features.md` and the
`examples/jdbc-v2-json-processors` example updated accordingly.

Server behaviour, isolated with `curl` — same query, same header, two servers:

```
$ curl -H 'X-ClickHouse-Format: TabSeparated' ... --data-binary "SELECT 1 AS a FORMAT JSONEachRow"
26.6.2.160: {"a":1} <- FORMAT clause wins
26.8.1.1761: 1 <- header wins
```

With no header at all, 26.8.1.1761 returns `{"a":1}`, so `FORMAT JSONEachRow` itself is still
honoured by the server; only the client's unconditional header suppresses it.

### Code Example

```java
try (QueryResponse response =
client.query("SELECT 1 AS a, 'x' AS b FORMAT JSONEachRow", new QuerySettings()).get()) {
System.out.println(response.getFormat());
// read response.getInputStream()
}
```

Observed against 26.8.1.1761:

```
response.getFormat() = RowBinaryWithNamesAndTypes
raw bytes (hex) = 02 01 61 01 62 05 55 49 6e 74 38 06 53 74 72 69 6e 67 01 01 78
```

That is `RowBinaryWithNamesAndTypes` (2 columns; `a`/`b`; `UInt8`/`String`; values 1, "x").
Expected `{"a":1,"b":"x"}`.

## Root cause

* `client-v2/src/main/java/com/clickhouse/client/api/Client.java`, `query(...)` — when the
caller sets no format, the request settings are defaulted:
`requestSettings.setFormat(ClickHouseFormat.RowBinaryWithNamesAndTypes)`. The SQL text is
never inspected for a `FORMAT` clause.
* `client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java`,
`addHeaders(...)` — that setting is emitted unconditionally as
`X-ClickHouse-Format` (`ClickHouseHttpProto.HEADER_FORMAT`).

Before 26.8 the two disagreeing instructions were harmless because the clause won. From 26.8
the header wins, so the default silently overrides the caller.

`jdbc-v2` inherits this: `StatementImpl.executeQueryImpl` branches on `response.getFormat()`,
which is read back from the response header, so its `JSONEachRow` and text-format branches
have become unreachable.

## Suggested fix

Make the header and the SQL agree, whichever direction the project prefers:

1. **Keep honouring the clause.** `ClickHouseSqlStatement.getFormat()` already extracts a
`FORMAT` clause; feed it into the request settings so the header matches what the caller
asked for. This preserves the Javadoc contract, the existing tests, and the `jdbc-v2` JSON
support, and needs no user-visible change.
2. **Deprecate the clause.** Detect a `FORMAT` clause in the caller's SQL and fail with a
clear error pointing at `QuerySettings.setFormat(...)`. This matches the direction stated
on #3068 ("format should be sent via settings not in query"), but it is a breaking change
and requires updating the Javadoc, the three tests above, `docs/client-v2-json-support.md`,
`docs/features.md` and `examples/jdbc-v2-json-processors`.

What must not remain is the current state, where the client accepts the clause and quietly
returns a different format.

Contrast case that must keep working: a caller who sets the format only through
`QuerySettings.setFormat(...)` and writes no `FORMAT` clause is unaffected today and must stay
unaffected. Inserts and client-v1 already keep header and clause in agreement; r2dbc sends no
format header.

## Configuration

#### Client Configuration
```java
Client client = new Client.Builder()
.addEndpoint(Protocol.HTTP, host, 8123, false)
.setUsername("default").setPassword(password)
.build();
```

#### Environment
* Client version: 0.10.0-rc1-SNAPSHOT, built from `main` at 40464dde1
* Language version: Java 17
* OS: Ubuntu 24.04 (Docker)

#### ClickHouse Server
* ClickHouse Server version: 26.8.1.1761 (`clickhouse/clickhouse-server:head`) reproduces;
26.6.2.160 does not
* ClickHouse Server non-default settings, if any: none
* `CREATE TABLE` statements for tables involved: none, `SELECT 1 AS a, 'x' AS b` is enough
* Sample data for all these tables: none needed

---

Found by automated analysis of clickhouse-java while working on #3068, and verified against a
live 26.8.1.1761 server and a 26.6.2.160 control rather than by inspection.

Contributor guide

Open the contributing guide

Research direction

Start with client-v2/src/main/java/com/clickhouse/client/api/Client.java, HttpAPIClientHelper.java, and ClickHouseSqlStatement.getFormat(), then run the three named client-v2 and jdbc-v2 tests against the stated server versions. Done means the project chooses and documents one explicit behavior for caller-supplied FORMAT clauses, preserves settings-only formats, and removes the silent mismatch in the affected tests and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.