ClickHouse / ClickHouse/clickhouse-js
[rowbinary] readRows/streamRowBatches silently drop all rows when every column is zero-width (Tuple())
- Dominant language
- TypeScript
- Stars
- 331
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
## Description
`@clickhouse/rowbinary` (`skills/clickhouse-js-node-rowbinary`) infers "no more rows" from "no more bytes":
```ts
// skills/clickhouse-js-node-rowbinary/src/readers/rows.ts:46
while (state.pos < state.buf.length) {
const row = readRow(state);
...
}
```
`Tuple()` occupies **zero bytes** in RowBinary. A result set whose columns are *all* zero-width therefore has zero-byte rows, and the cursor-vs-buffer-length probe cannot distinguish N such rows from end-of-stream. Every row is dropped **silently** — no exception, no warning, `readRows` returns `[]`. `streamRowBatches` (`src/readers/stream.ts`) is built on `readRows`, so the streaming path silently yields nothing too.
Server evidence (the wire really is 0 bytes for 3 rows, so the count is not recoverable from the byte stream):
```
$ curl -s 'http://localhost:8123/?query=SELECT tuple() FROM numbers(3) FORMAT RowBinary' | wc -c
0
$ curl -s 'http://localhost:8123/?query=SELECT count() FROM (SELECT tuple() FROM numbers(3))'
3
```
Contrast case that already works and must not regress: as soon as one non-zero-width column is present (`SELECT tuple(), toUInt8(5)`), rows have non-zero byte length and the driver is correct.
Related, and currently masking the above on the header-driven path: **`Tuple()` cannot be compiled at all**. `astToReader` routes the empty tuple into `dataTypeReader`, which falls through to the `NULLARY` lookup and throws `RowBinaryTypeError: unsupported RowBinary type: Tuple` (`src/readers/compile.ts:204`) — even when a normal column sits next to it. So `compileRowBinaryWithNamesAndTypes` throws rather than under-reporting; the silent loss is reachable through hand-written / codegen'd row readers, which is the package's primary documented usage.
The main clients (`@clickhouse/client`, `@clickhouse/client-web`) are **not** affected: they read newline-framed text formats (`JSONEachRow` etc.), where `SELECT tuple() FROM numbers(3)` yields three `{"()":[]}` lines.
## ClickHouse server version
26.7.3.19 (verified against a running server at `http://localhost:8123`).
## Reproduction
Vitest, run from `skills/clickhouse-js-node-rowbinary` (uses the package's own `tests/clickhouse.ts` helper):
```ts
import { describe, expect, it } from "vitest";
import { query } from "./clickhouse.js";
import { Cursor } from "../src/readers/core.js";
import { readRows } from "../src/readers/rows.js";
import { readTuple } from "../src/readers/composite.js";
import { streamRowBatches } from "../src/readers/stream.js";
import { compileRowBinaryWithNamesAndTypes } from "../src/readers/rowBinaryWithNamesAndTypes.js";
describe("zero-width (Tuple()) rows", () => {
it("readRows drops every zero-width row", async () => {
const buf = await query("SELECT tuple() FROM numbers(3) FORMAT RowBinary");
expect(buf.length).toBe(0); // 3 rows, 0 bytes on the wire
const rows = readRows(readTuple([]))(new Cursor(buf));
expect(rows.length).toBe(3); // actual: 0
});
it("streamRowBatches over the same response yields nothing", async () => {
const buf = await query("SELECT tuple() FROM numbers(3) FORMAT RowBinary");
async function* chunks() { yield buf }
let rows = 0;
for await (const batch of streamRowBatches(chunks(), readTuple([]), { warnOnSmallChunks: false })) {
rows += batch.length;
}
expect(rows).toBe(3); // actual: 0
});
it("compiled header path throws on Tuple()", async () => {
const buf = await query(
"SELECT tuple() AS t, toUInt8(5) AS n FROM numbers(3) FORMAT RowBinaryWithNamesAndTypes",
);
const s = new Cursor(buf);
const compiled = compileRowBinaryWithNamesAndTypes(s); // throws
expect(compiled.readRows(s).length).toBe(3);
});
});
```
Actual output:
```
❯ tests/ZeroWidth.test.ts (3 tests | 3 failed)
× readRows drops every zero-width row
AssertionError: expected +0 to be 3
× streamRowBatches over the same response yields nothing
AssertionError: expected +0 to be 3
× compiled header path throws on Tuple()
RowBinaryTypeError: unsupported RowBinary type: Tuple
❯ dataTypeReader src/readers/compile.ts:204:13
```
Expected: 3 rows (each `[]` / `{ t: [], n: 5 }`) in all three cases.
## Suggested fix
- `src/readers/compile.ts` — route an argument-less `Tuple` to `tupleReader` (or add a zero-width leaf) so `Tuple()` / `Tuple(Tuple())` compile at all; a zero-column tuple reader consumes 0 bytes and returns `[]`.
- `src/readers/rows.ts` / `src/readers/stream.ts` — the row boundary cannot come from byte presence when the compiled row width is a fixed 0. Options, in rough order of intrusiveness (a maintainer design call):
1. Detect the degenerate case (every column reader is fixed zero-width) and source the row count out of band — e.g. the response's `X-ClickHouse-Summary` header (`read_rows`), or document that callers must supply it.
2. Throw on a zero-width row reader rather than silently returning `[]` — worse than (1) for users, far better than silent data loss.
3. Use a framed format (`Native` block headers carry the row count) for such result sets.
- Whatever the choice, note it in the `readRows` docstring, which currently states the byte-exhaustion invariant as unconditional.
## Link
Reported for the .NET client as ClickHouse/clickhouse-cs#570 (same defect class: end-of-rows inferred from end-of-bytes). Found by automated cross-client analysis; verified here against ClickHouse 26.7.3.19.
Contributor guide
Research direction
Start with tests/ZeroWidth.test.ts, then read src/readers/rows.ts, src/readers/stream.ts, and src/readers/compile.ts. Trace how zero-width readers and end-of-buffer detection behave, including the RowBinaryWithNamesAndTypes entry point. Done means the chosen handling is covered by tests, Tuple() compiles where intended, and zero-width results no longer disappear silently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100