ClickHouse / ClickHouse/clickhouse-js
Uncompressed streaming inserts emit one HTTP chunk per row, inflating the request body ~50%
- Dominant language
- TypeScript
- Stars
- 331
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
## Description
When `insert` is given an **object-mode stream** (or a raw-format stream that pushes small pieces) and request compression is **off** (the default — `compression.request` defaults to `undefined`, see `packages/client-common/src/config.ts:393` + `normalizeRequestCompression`), each row reaches the transport as its own `write()` on the `http.ClientRequest`. The body is sent with `Transfer-Encoding: chunked`, so every row becomes its own HTTP chunk: `"\r\n" + row + "\r\n"`, i.e. ~5 extra wire bytes and one write syscall per row.
Path:
- `packages/client-node/src/utils/encoder.ts:26-32` — object-mode values are piped through `mapStream(...)`,
- `packages/client-node/src/utils/stream.ts:49-58` — that `Transform` pushes **one string per row**,
- `packages/client-node/src/connection/socket_pool.ts:395-402` — `pipeStream()` pipes that stream straight into `request` when `params.request_compression` is unset. With compression enabled the gzip/zstd `Transform` sits in between and coalesces writes into large blocks, which is why this is invisible on the compressed path.
Array inputs are unaffected: `encodeValues` joins all rows into a single string (`encoder.ts:34-37`), so the body is one chunk.
Measured on a 1000-row `JSONEachRow` insert of `{x: i}` (raw wire bytes captured from a TCP sink, so chunk framing is real, not estimated):
| input | request compression | payload | body wire bytes | chunks | overhead |
|---|---|---|---|---|---|
| array of 1000 objects | off | 9,890 B | 9,903 B | 1 | +0.1% |
| object-mode stream, 1000 rows | off | 9,890 B | **14,895 B** | **1000** | **+50.6%** |
| object-mode stream, 1000 rows | gzip | 9,890 B | 2,000 B | 2 | +0.9% |
The inflation scales with how small a row is: narrow rows pay the most, and a wide row still pays ~5 B plus one extra socket write. This is the same defect reported for the .NET client in ClickHouse/clickhouse-cs#524 (there it is one chunk per *field* on the uncompressed `RowBinary` path); in `clickhouse-js` it is one chunk per *row*, and unlike the .NET client the affected path is the **default** one, since request compression is off by default here.
## ClickHouse server version
`26.7.2.59` (the wire measurement itself uses a local TCP sink rather than the server, so the request bytes are captured verbatim; a real server accepts the body either way — this is purely a wire-efficiency issue, not a correctness one).
## Reproduction
Self-contained script (run from the repo root with `npx tsx`); it points the client at a local TCP sink and counts chunked-encoding frames in the captured request body.
```ts
import net from "net";
import Stream from "stream";
import { createClient } from "./packages/client-node/src/index.js";
function analyze(raw: Buffer) {
const sep = raw.indexOf("\r\n\r\n");
const headers = raw.subarray(0, sep).toString();
const body = raw.subarray(sep + 4);
if (!/transfer-encoding: chunked/i.test(headers)) {
return { chunks: 1, payload: body.length, wire: body.length };
}
let off = 0, chunks = 0, payload = 0;
while (off < body.length) {
const nl = body.indexOf("\r\n", off);
if (nl < 0) break;
const size = parseInt(body.subarray(off, nl).toString(), 16);
if (Number.isNaN(size) || size === 0) break;
chunks++;
payload += size;
off = nl + 2 + size + 2;
}
return { chunks, payload, wire: body.length };
}
async function run(label: string, values: () => any, port: number, gzip = false) {
const captured: Buffer[] = [];
const server = net.createServer((socket) => {
socket.on("data", (d) => {
captured.push(d);
if (Buffer.concat(captured).includes("0\r\n\r\n")) {
socket.write("HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
socket.end();
}
});
});
await new Promise((r) => server.listen(port, r));
const client = createClient({
url: `http://localhost:${port}`,
compression: { request: gzip },
});
try {
await client.insert({ table: "t", values: values(), format: "JSONEachRow" });
} catch {
/* the sink returns a minimal response */
}
await client.close();
server.close();
const a = analyze(Buffer.concat(captured));
console.log(
`${label}: chunks=${a.chunks} payload=${a.payload} wire=${a.wire} ` +
`overhead=${(((a.wire - a.payload) / a.payload) * 100).toFixed(1)}%`,
);
}
async function main() {
const rows = Array.from({ length: 1000 }, (_, i) => ({ x: i }));
await run("array", () => rows, 18201);
await run("stream", () => Stream.Readable.from(rows, { objectMode: true }), 18202);
await run("stream + gzip", () => Stream.Readable.from(rows, { objectMode: true }), 18203, true);
}
main();
```
Expected — the streaming insert should cost roughly what the array insert costs (a handful of chunks, ~0% framing overhead):
```
array: chunks=1 payload=9890 wire=9903 overhead=0.1%
stream: chunks=~1 payload=9890 wire=~9903 overhead=~0.1%
stream + gzip: chunks=2 payload=9890 wire=2000 overhead=0.9%
```
Actual:
```
array: chunks=1 payload=9890 wire=9903 overhead=0.1%
stream: chunks=1000 payload=9890 wire=14895 overhead=50.6%
stream + gzip: chunks=2 payload=9890 wire=2000 overhead=0.9%
```
## Suggested fix
Give the uncompressed path the same coalescing the compressed path gets for free — insert a buffering stage between the body stream and the request in `packages/client-node/src/connection/socket_pool.ts:395-402`, e.g. pipe through a `Transform` that accumulates into a buffer (the compressors effectively use tens of KB) and flushes on threshold and on `final`, so:
```ts
Stream.pipeline(bodyStream, coalesce(), request, callback);
```
Alternatively (or additionally) batch inside `mapStream` (`packages/client-node/src/utils/stream.ts:49-58`) so N encoded rows are pushed as one string. Buffering at the connection level is the more general fix, since it also helps raw-format streams (CSV/TSV/RowBinary) whose producer pushes small pieces. Whichever place it lands, the flush on stream end must happen before `request.end()` or the tail of the batch is lost.
## Link
Relayed from ClickHouse/clickhouse-cs#524 — same root cause (no buffer between the row serializer and the chunked HTTP request stream on the uncompressed path).
Contributor guide
Research direction
Start at packages/client-node/src/connection/socket_pool.ts:395-402 and trace the uncompressed pipeStream path back through packages/client-node/src/utils/encoder.ts:26-37 and packages/client-node/src/utils/stream.ts:49-58. Run the provided npx tsx reproduction to capture chunk counts, then verify that uncompressed streaming inserts coalesce rows into a handful of chunks while compressed and array inputs remain working.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100