ClickHouse / ClickHouse/clickhouse-js

In-band mid-stream exceptions are not detected for exec() streams (Arrow/Native) or ResultSet.text(): callers get a raw ECONNRESET "aborted" instead of the server error

Open
#976 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
331
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Description

When a query fails **after** ClickHouse has already committed `200 OK` and started streaming the body (e.g. a runtime `throwIf` partway through a large result), the server (25.11+) appends an in-band exception block, announced by the `x-clickhouse-exception-tag` response header and delimited by `__exception__` markers.

`@clickhouse/client` only inspects that in-band block in **one** place: the newline-splitting transform inside `ResultSet.stream()` (`packages/client-node/src/result_set.ts:252-257`, `packages/client-web/src/result_set.ts:200-205`), which calls `extractErrorAtTheEndOfChunk`. Every other way of consuming a response body skips it:

1. **`client.exec()`** — this is the documented way to fetch binary/columnar output (see [`examples/node/performance/select_parquet_as_file.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/performance/select_parquet_as_file.ts), which pipes the `exec()` stream straight to a file). `exec()` returns the transport stream untouched — nothing ever looks at `x-clickhouse-exception-tag`, even though the header is present on the response. A mid-stream failure surfaces as `Error: aborted` / `ECONNRESET`, and any already-written output file silently contains a truncated payload plus the raw `__exception__` trailer bytes.
2. **`ResultSet.text()`** — reads the whole body via `getAsText()` with no exception check, so the same query that produces a clean `ClickHouseError` through `.stream()` produces `Error: aborted` (`ECONNRESET`) through `.text()`.

This is the Node.js/Web analogue of https://github.com/ClickHouse/clickhouse-connect/issues/913 (Arrow streaming methods bypassing in-band exception detection).

Note: streaming formats that ClickHouse buffers fully server-side (e.g. `Parquet`) are *not* affected — the error arrives before any body bytes, so the normal error path fires. The problem is specific to formats that actually stream (`Arrow`, `Native`, `RowBinary`, `CSV`, ...).

## ClickHouse server version

`26.7.1.1315` (local, verified end-to-end).

## Reproduction

`packages/client-node/__tests__/integration/inband_exception_exec.test.ts`:

```ts
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import type { ClickHouseClient } from "@clickhouse/client-common";
import { createTestClient } from "@test/utils";
import type Stream from "stream";

const QUERY =
"SELECT number, throwIf(number = 1000000, 'boom') FROM system.numbers";

describe("in-band exception on non-row-parser paths", () => {
let client: ClickHouseClient;
beforeEach(() => {
client = createTestClient() as ClickHouseClient;
});
afterEach(async () => {
await client.close();
});

// PASSES today - the row parser detects the in-band exception.
it("stream() surfaces the server error", async () => {
const rs = await client.query({ query: QUERY, format: "JSONEachRow" });
await expect(
(async () => {
for await (const rows of rs.stream()) void rows;
})(),
).rejects.toMatchObject({ code: "395" });
});

// FAILS today.
it("exec() with FORMAT Arrow surfaces the server error", async () => {
const { stream, response_headers } = await client.exec({
query: `${QUERY} FORMAT Arrow`,
});
expect(response_headers["x-clickhouse-exception-tag"]).toBeDefined(); // header IS there
await expect(
(async () => {
for await (const chunk of stream) void chunk;
})(),
).rejects.toMatchObject({ code: "395" });
});

// FAILS today.
it("text() surfaces the server error", async () => {
const rs = await client.query({ query: QUERY, format: "CSV" });
await expect(rs.text()).rejects.toMatchObject({ code: "395" });
});
});
```

Observed (instrumented run against 26.7.1.1315):

```
JSONEachRow stream(): ClickHouseError code=395 "boom: while executing 'FUNCTION throwIf(...)'" <- correct
exec() FORMAT Arrow: 3934840 bytes consumed, then Error code=ECONNRESET msg="aborted" <- no server error
exec() FORMAT Native: 8831550 bytes consumed, then Error code=ECONNRESET msg="aborted" <- no server error
CSV text(): Error code=ECONNRESET msg="aborted" <- no server error
exec() FORMAT Parquet: 0 bytes, ClickHouseError code=395 <- fine (buffered server-side)
```

Expected: all four raise `ClickHouseError` with `code: '395'` and the `boom` message, matching the `stream()` path.

## Suggested fix

The building blocks already exist — `extractErrorAtTheEndOfChunk` and `EXCEPTION_TAG_HEADER_NAME` in `packages/client-common/src/utils/stream.ts` — they are just not applied outside the row parser:

- Wire a format-agnostic in-band exception detector into the stream returned by `ClickHouseClient.exec()` when `x-clickhouse-exception-tag` is present on the response (`packages/client-common/src/client.ts:420`, plus the node/web connection `exec` implementations). The passthrough detector proposed in #778 (`RawStreamExceptionDetector`) looks like the right primitive; #778 only wires it into `ResultSet.binaryStream()` for `query()` results, so `exec()` would still be uncovered.
- Apply the same check in `ResultSet.text()` (`packages/client-node/src/result_set.ts:157`, `packages/client-web/src/result_set.ts`) — currently `getAsText()` bypasses it entirely.

Related but not covering these paths: #778 (`binaryStream()` for `query()`), #803 (opt-in transformer for the Node `ResultSet` row parser), #974 (false positives in the current `\r\n` heuristic).

## Link

Relayed from https://github.com/ClickHouse/clickhouse-connect/issues/913
Central tracking issue: https://github.com/ClickHouse/integrations-ai-playground/issues/330

Contributor guide

Open the contributing guide

Research direction

Start with packages/client-common/src/utils/stream.ts and the exec implementations referenced around packages/client-common/src/client.ts:420, then compare the existing ResultSet.stream() handling. Inspect packages/client-node/src/result_set.ts:157 and its web counterpart, and run packages/client-node/__tests__/integration/inband_exception_exec.test.ts. Done means exec() and ResultSet.text() surface ClickHouseError code 395 for the streaming reproductions without regressing the existing stream() path.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.