Discussion: pluggable Connection design for chDB integration in @clickhouse/client

## TL;DR

To let `@clickhouse/client` users target either a remote ClickHouse server or in-process chDB **with a one-line change** — without forking the client, without losing chDB's zero-copy advantage, and without making `@clickhouse/client` carry chDB-specific code — this proposal introduces a **pluggable `Connection`** behind a single public client API:

- `@clickhouse/client` exposes a public `Connection` interface (it already has private `HttpConnection`/`HttpsConnection` — promote them to a public contract).
- `createClient()` accepts an injected `connection` option.
- **chdb-node** ships a `ChdbConnection` implementation that consumes its existing Layer 1 native session.
- **`@clickhouse/client` ships zero chDB code, zero chDB tests, zero chDB CI dependency.**
- A `supportsZeroCopyStreaming` capability flag lets the client route streaming through chDB's in-process record-batch path when available, preserving end-to-end zero-copy.
- chDB-unique surface (`Python()` table function, UDF registration, in-memory vs persistent path management) lives in a typed extension namespace (`client.chdb.*`), available only when the chdb connection is in use.
- Tests run on every installed connection via a parameterized fixture; **chDB owns the skip manifest** in its own repo via a published JSON profile.
- CI is split: `@clickhouse/client` CI never touches chDB; chdb-node CI runs `@clickhouse/client`'s test suite against `ChdbConnection`.

## Core principle (one sentence)

> **`@clickhouse/client` provides "one `ClickHouseClient` + a public `Connection` interface + a `connection` injection point on `createClient()`"; chdb-node provides a `ChdbConnection` implementation consumed by user code via that injection point. The two packages are coupled only through the `Connection` interface — `@clickhouse/client` ships zero chDB code, zero chDB tests, and zero chDB CI dependency.**

---

The detailed design is folded below — expand the sections you want to dig into.

📖 1. Background — what users want, and two tempting approaches that this proposal rejects

### 1.1 What users want

Two recurring asks from the chDB and `@clickhouse/client` user communities:

1. **One-line switch between remote ClickHouse and in-process chDB** — a developer writing code against `@clickhouse/client` wants to point that same code at chDB for local development, CI tests, ephemeral environments, or embedded analytics, without rewriting any business logic.
2. **No compromise on the embedded engine's advantage** — chDB's value over a remote server is in-process zero-copy data exchange. If the integration loses that, users have no reason to choose chDB over a local ClickHouse container.

A third constraint comes from the `@clickhouse/client` maintainers: the public client surface is intentionally slim, and bolting a second client family onto it (one full surface for HTTP, another for chDB) is a maintenance trap — every new public method, every parameter change, every streaming or settings refactor would have to be applied twice.

### 1.2 Two tempting approaches that this proposal rejects

**Approach A — A parallel "compat client" for chDB.** Ship a separate `ChdbClickHouseClient` class that mirrors `@clickhouse/client`'s public surface and translates calls onto chDB's native API. This is byte-compatible and works, but:

- Every public method, type, format, settings handler, and error class lives twice. Upstream changes drift silently.
- Preserving byte-compatibility tends to force eager-buffered semantics for `query()` and `stream()` (so errors surface where users expect them), which **erases the zero-copy advantage** that justifies using chDB in the first place.
- The translation glue grows linearly with the public surface — easily 1,500+ LOC of types, params, settings, formats, URL parsing, error mapping, and result reshaping that exists only to mirror upstream shapes.

**Approach B — A loopback HTTP server inside chdb-node.** Expose chDB through a localhost-only HTTP endpoint and let `@clickhouse/client`'s existing `HttpConnection` consume it. Minimal JS-side code, but:

- Two serialization edges (chDB columns → wire format; wire format → JS objects), plus HTTP chunk framing, plus an extra buffer copy.
- chDB's native record-batch streaming becomes inaccessible behind a socket.
- chdb-node's embedded path is deliberately networkless (footprint, port management, lifecycle, single-process security model).

Both approaches trade chDB's reason for existing for ergonomics. This proposal proposes a third path that keeps the ergonomics and preserves the embedded advantage.

📦 2. Repository boundary — what lives where, and what is the only coupling surface

```
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐
│ @clickhouse/client repo │ │ chdb-io/chdb-node repo │
│ github.com/ClickHouse/clickhouse-js │ │ github.com/chdb-io/chdb-node │
│ (owned by CH team) │ │ (owned by chDB team) │
├──────────────────────────────────────────┤ ├──────────────────────────────────────────┤
│ src/ │ │ src/ │
│ client.ts public semantics │ │ connection.ts ChdbConnection │
│ connection.ts Connection iface[new]│ │ extension.ts .chdb extension │
│ http_connection.ts HTTP impl │ │ test_profile.ts skip manifest │
│ https_connection.ts HTTPS impl │ │ package.json │
│ │◄───┤ "exports": { │
│ createClient(opts: { connection? }) │ │ "./connection": │
│ │ │ "./dist/connection.js" │
│ test/ │ │ } │
│ fixtures/ parameterized over conns │ │ test/ │
│ contract/ Connection contract suite │ │ runs full @clickhouse/client suite │
│ │ │ against ChdbConnection │
│ ✗ zero chdb code │ │ ✓ CI installs chdb-node + clickhouse-js │
│ ✗ zero chdb tests │ │ ✓ test_profile.json is authoritative │
│ ✗ CI does not depend on chdb-node │ │ │
└──────────────────────────────────────────┘ └──────────────────────────────────────────┘
▲ │
│ interface signature + named exports │
└──────────────────────────────────────────────┘
(the only coupling surface)
```

The coupling surface is intentionally tiny:

- The exported `Connection` TypeScript interface signature.
- One package subpath export: `chdb-node/connection`.

Everything else — chDB version, Layer 1 native API quirks, output format details, FFI shape, filesystem paths — is invisible to `@clickhouse/client`. The `ChdbConnection` adapter consumes chdb-node's already-public Layer 1 API (`Session`, `queryAsync`, `streamQuery`, record-batch reader) and does **not** require any changes to chdb-core.

🔧 3. Refactor and abstraction — before/after, plus the interface-evolution discipline

### Before

```
ClickHouseClient
├─ Public API: query / query_arrow / insert / exec / command / ping / close
├─ Semantics: ResultSet, type handling, settings, error model, format negotiation
└─ Transport: HttpConnection / HttpsConnection (private internal classes)
```

`ClickHouseClient` is the only client; transport is selected internally based on URL scheme but cannot be replaced by a user.

### After

```
ClickHouseClient ← single semantics layer, shared by all connections
├─ query / query_arrow / insert / exec / command / ping / close
├─ ResultSet / type handling / format negotiation
├─ settings normalization, unified errors
└─ this.connection: Connection ◄─┐
│ injected by createClient({ connection })
┌──────────────────────────┴──────────────────┐
│ │
HttpConnection ChdbConnection
(@clickhouse/client repo) (chdb-node repo)
- query / insert / command via HTTP - query via chdb Session.queryAsync()
- undici / agentkeepalive pool - query stream: native record-batch reader
- supportsZeroCopyStreaming = false via Layer 1 streamQuery
(HTTP wire bytes always cross a - insert via Python() table function for
socket boundary; streaming gains are zero-copy DataFrame ingest where the
bounded by network framing) caller passes a typed array
- supportsZeroCopyStreaming = true
```

Transport is extracted from the client's internal selection logic into an explicit `connection` injection. Public semantics, type handling, settings, the error model, and the async/Promise surface all stay in one shared layer.

### Interface-evolution discipline

To prevent the `Connection` interface from churning and breaking registered connections:

1. **Methods are additive only** — never remove, never change a method signature.
2. **Every new method ships with a default implementation** that falls back to existing primitives, so old connections keep working without code changes.
3. **Capabilities are declared via `supports*` flags** and are additive — flag meanings are never repurposed; new capabilities get new flag names.

With this discipline, interface evolution is a one-way ratchet: `@clickhouse/client` can add anything, but cannot reach back and break an already-shipped connection.

🧬 4. API design — Connection interface, capability negotiation, .chdb extension namespace

### 4.1 Connection interface (defined in `@clickhouse/client`)

```ts
// @clickhouse/client/src/connection.ts (public export)

export interface Connection {
// ── Identity ──────────────────────────────────────────────
readonly connectionName: string

// ── Capability flags ──────────────────────────────────────
// Routing flag with real semantic impact today:
// true → ClickHouseClient.query streaming should prefer the
// native record-batch path (genuinely zero-copy in-process)
// false → ClickHouseClient.query streaming should keep using
// the chunked-bytes path
readonly supportsZeroCopyStreaming: boolean

// ── Core transport (mandatory) ────────────────────────────
query(params: QueryParams): Promise
queryStream(params: QueryParams): AsyncIterable
insert(params: InsertParams): Promise
exec(params: ExecParams): Promise
command(params: CommandParams): Promise
ping(): Promise
close(): Promise

// ── Error mapping (connection-specific errors never leak) ─
mapError(err: unknown): ClickHouseError

// ── Server metadata ───────────────────────────────────────
readonly serverVersion: Promise
readonly serverTimezone: Promise
}

export interface QueryResult {
stream: AsyncIterable
summary: ClickHouseSummary
queryId: string
}
```

| Connection | `supportsZeroCopyStreaming` | Why |
|---|---|---|
| `HttpConnection` | `false` | HTTP wire bytes always cross a socket; streaming gains are bounded by chunked-transfer framing. The client's existing chunked-bytes path is the right model. |
| `HttpsConnection` | `false` | Same as HTTP. |
| `ChdbConnection` | `true` | chDB exposes an in-process native record-batch reader via Layer 1's `streamQuery`. The client routes streaming through it directly, zero-copy. |

### 4.2 Capability negotiation in `ClickHouseClient` (this is where zero-copy is preserved)

```ts
// @clickhouse/client/src/client.ts
async queryStream(params: QueryParams): Promise> {
if (this.connection.supportsZeroCopyStreaming) {
// in-process; native record-batch reader; no socket; no wire format
return this.connection.queryStream(params)
}
// existing chunked-bytes streaming path; behavior unchanged for HTTP users
return this.#chunkedStream(params)
}
```

The capability flag controls only the **routing decision inside the streaming entry points**. On chDB the native path is strictly better (zero-copy); on HTTP it isn't (no socket-side benefit). For the materialized-row paths (`query().then(rs => rs.json())`) both connections converge through `queryStream` and the same row-shaping code.

### 4.3 `.chdb` extension namespace (for chDB-unique APIs)

chDB has capabilities that have no equivalent on a remote ClickHouse server. Rather than adding optional methods to `ClickHouseClient`, they surface via a typed extension that is **only present when the chdb connection is in use**:

```ts
import { createClient } from '@clickhouse/client'
import { createChdbConnection } from 'chdb-node/connection'

const client = createClient({ connection: createChdbConnection({ path: ':memory:' }) })

// chdb-only surface, typed:
await client.chdb.queryPython({ sql, df: myDataFrame }) // Python() table function
client.chdb.registerFunction(myUdf) // UDF registration
client.chdb.sessionPath // in-memory vs persistent path
client.chdb.cursor() // native DB-API cursor (escape hatch)
```

The typing trick (TypeScript discriminated brand on the connection) makes `client.chdb` exist in the type system **only** when the connection is `ChdbConnection`:

```ts
// In @clickhouse/client:
type ConnectionBrand = C extends { __brand: infer B } ? B : 'http'
type ChdbSurface = ConnectionBrand extends 'chdb' ? { chdb: ChdbExtension } : {}
type ClickHouseClient = BaseClient & ChdbSurface

// In chdb-node:
export interface ChdbConnection extends Connection { readonly __brand: 'chdb' }
```

Accessing `client.chdb` on an HTTP connection is a TypeScript error at compile time and a runtime `undefined`. The `ChdbExtension` interface and its implementation live in the chdb-node repo (`src/extension.ts`); `@clickhouse/client` provides only the type-level brand check.

👤 5. User experience — installation and one-line switch

```ts
import { createClient } from '@clickhouse/client'

// Default HTTP — existing users: zero changes, full forward compatibility.
const client = createClient({
url: 'https://prod.clickhouse.cloud',
username: '...',
password: '...',
})

// Switch to chDB in-process — one line.
import { createChdbConnection } from 'chdb-node/connection'
const client = createClient({
connection: createChdbConnection({ path: ':memory:' }),
})

// All downstream code is identical across connections.
const rs = await client.query({ query: 'SELECT * FROM events LIMIT 100' })
for await (const row of rs.stream()) { /* ... */ }
await client.insert({ table: 'events', values: rows })
```

**Installation:**

```bash
npm install @clickhouse/client # existing users, unchanged
npm install @clickhouse/client chdb-node # users who want chDB add chdb-node
```

After `chdb-node` is installed, importing `chdb-node/connection` is the explicit opt-in. The `@clickhouse/client` bundle size, dependency tree, and TypeScript declarations are unchanged for users who don't install chdb-node.

If a user passes a `connection` instance built against an incompatible `Connection` interface version, `createClient()` throws a clear `IncompatibleConnectionError` pointing at the install command and the supported interface version.

> **Note on registration mechanism.** This proposal uses **explicit dependency injection** via the `connection` option — no global registries, no side-effect imports, easier to reason about under tree-shaking and bundlers. The trade-off is that the user writes one extra `import` line; in exchange they get a fully typed, explicit configuration with no runtime magic.

📋 6. Code-change responsibility matrix — who changes what, when

| Scenario | @clickhouse/client changes? | chdb-node changes? |
|---|---|---|
| chDB engine upgrade (new version, perf, bug fix) | ❌ no | ✅ `connection.ts` updated |
| chDB output-format behavior change | ❌ no | ✅ `connection.ts` adapted |
| CH server adds a new public feature → `ClickHouseClient` gains a new API | ✅ add `client.foo()` (default impl on `Connection` falls back to `query`) | usually no; optionally implement a fast path in `ChdbConnection` |
| `HttpConnection` perf / pool tweaks | ✅ `HttpConnection` | ❌ no |
| chDB adds a unique capability (new UDF kind, new Python integration) | ❌ no | ✅ extend `.chdb` namespace in `src/extension.ts` |
| Test discovers chDB doesn't support a CH feature | ❌ no | ✅ add a marker to `test_profile.SKIP_MARKERS` |
| `Connection` interface itself needs a new method | ✅ add method + default impl + capability flag | optionally implement optimized path |
| User reports buggy SQL behavior on chDB | ❌ no | ✅ chdb-node team investigates |
| User reports buggy behavior on HTTP | ✅ CH team investigates | ❌ no |

**Invariant: a chDB upstream change never forces a `@clickhouse/client` change.**

🧪 7. Testing architecture — parameterized fixtures, marker hygiene, skip manifest, CI split, contract suite

### 7.1 Parameterized fixtures — write once in `@clickhouse/client`

```ts
// @clickhouse/client/test/fixtures.ts
import { describe, beforeEach } from 'vitest'
import { createClient } from '@clickhouse/client'

const connections = ['http']
try {
const { createChdbConnection } = await import('chdb-node/connection')
connections.push('chdb')
} catch { /* chdb-node not installed; HTTP-only run */ }

export function describeForEachConnection(name: string, body: (factory: () => ClickHouseClient) => void) {
for (const conn of connections) {
describe(`${name} [${conn}]`, () => body(() => makeClient(conn)))
}
}
```

**This fixture does not let chdb-node affect `@clickhouse/client`'s CI** — the dynamic `import()` either succeeds (chdb-node is installed → adds the `[chdb]` parameter) or throws and is silently caught (chdb-node is not installed → fixture only enumerates `['http']`):

- In the `@clickhouse/client` repo's CI, **chdb-node is not installed** (`package.json` `dependencies` / `devDependencies` / workflow files don't reference it) → the dynamic import rejects → the fixture produces `['http']` only → tests fan out on http only, **behavior identical to today**.
- The same fixture in the chdb-node repo's CI (which **does** install chdb-node and `@clickhouse/client`) automatically picks up the `[chdb]` parameter dimension and runs the full suite against `ChdbConnection` — that's chdb-node's CI burden, and `@clickhouse/client` doesn't see it.

So CH developers writing new tests **do not need to know anything about chDB** — their tests fan out across connections for free, and CI behavior is unchanged.

### 7.2 Marker hygiene (the only new habit for CH developers)

Tag tests with the server feature they depend on. This is good test hygiene that pays for itself independently of chDB:

```ts
test.requiresFeature('distributed', async () => { /* ... */ })
test.requiresFeature('keeper', async () => { /* ... */ })
test.requiresFeature('replicated', async () => { /* ... */ })
test.requiresFeature('grants', async () => { /* ... */ })
test.requiresFeature('http_only', async () => { /* ... */ }) // compression negotiation, proxy
```

The feature set lives in `@clickhouse/client`'s test helpers and grows organically.

### 7.3 Skip manifest — owned by chdb-node

```jsonc
// chdb-node/dist/test_profile.json ── inside the chdb-node repo
{
"skipMarkers": [
"requires_distributed",
"requires_replicated",
"requires_keeper",
"requires_grants",
"requires_async_insert",
"http_only"
],
"skipNodeIds": {
"test/query.test.ts > complex codecs": "chdb does not yet support ALTER MODIFY CODEC"
}
}
```

`@clickhouse/client`'s test runner loads the profile via the package's subpath export and applies skips dynamically. This code is written **once** and never touched again.

**chdb-node owns data, not code.** The skip manifest is a declarative file maintained in the chdb-node repo.

### 7.4 CI matrix split

```
@clickhouse/client repo CI: chdb-node repo CI:
matrix: connection=[http] npm install chdb-node @clickhouse/client
runs against a real CH server vitest --pool=threads test/upstream
─ never fails because of chdb-node ─ runs the entire CH suite on chdb conn
─ never pins to a chdb-node version ─ chdb-node upstream changes surface here first
─ chdb-node team fixes connection or
extends skip manifest
```

The chdb-node repo also subscribes to `@clickhouse/client`'s release tags and re-runs the regression on each new CH release. **CH developers are transparent to this pipeline — they do nothing for it.**

### 7.5 Connection-contract suite (`@clickhouse/client` repo)

A small generic suite asserting invariants any connection must satisfy. Runs against the HTTP connection and a mock connection in CH's CI. The chdb-node repo runs the same suite against `ChdbConnection`. This is the safety net that guarantees connection behavior is predictable.

---

## Decoupling mechanisms — summary

```
┌──────────────────────────────────────────────────────────────────┐
│ 1. Connection interface (code contract) │
│ additive only + default impls + capability flags │
│ → interface evolution never breaks shipped connections │
├──────────────────────────────────────────────────────────────────┤
│ 2. Explicit DI via createClient({ connection }) │
│ no global registry; no side-effect imports │
│ → @clickhouse/client does not import chdb-node; │
│ chdb-node does not fork @clickhouse/client │
├──────────────────────────────────────────────────────────────────┤
│ 3. .chdb extension namespace (unique-API outlet) │
│ TS-typed brand check; runtime presence iff chdb connection │
│ → chDB-only capabilities do not pollute the base interface │
├──────────────────────────────────────────────────────────────────┤
│ 4. Skip profile (testing boundary as data) │
│ feature markers in CH, JSON manifest in chdb-node │
│ → CH writes tests without knowing chDB; chdb-node does not │
│ fork CH's test suite │
├──────────────────────────────────────────────────────────────────┤
│ 5. CI matrix split │
│ CH CI depends on zero chdb-node state │
│ → chDB upstream changes never turn CH CI red │
└──────────────────────────────────────────────────────────────────┘
```

---

## Next step

If this direction looks right to the maintainers, the staged landing plan would be:

1. **Promote `Connection` to a public interface** in `@clickhouse/client` (extract today's private shape, add the `connection?:` option to `createClient()`, default behavior byte-identical to today).
2. **Add the capability flag + extension brand** (`supportsZeroCopyStreaming`, the TS brand machinery for `.chdb`) — purely additive.
3. **Ship `ChdbConnection` in chdb-node** (`src/connection.ts`, `src/extension.ts`, `src/test_profile.ts`, the `./connection` subpath export, plus a chdb-node-side CI job that runs `@clickhouse/client`'s test suite against `ChdbConnection`).

Step 1 is the only step that touches production HTTP behavior; steps 2 and 3 are strictly additive on the `@clickhouse/client` side.

The same architectural shape is being discussed in parallel for the Python ecosystem at [ClickHouse/clickhouse-connect#809](https://github.com/ClickHouse/clickhouse-connect/issues/809); adopting it here keeps the two ecosystems aligned.

Happy to discuss any part of the proposal here or in a sync.

Start by reading src/client.ts, src/connection.ts, src/http_connection.ts, and src/https_connection.ts to understand the existing client and transport boundaries. Then inspect test/fixtures and test/contract for the current test structure. Done means the proposed public Connection contract and connection injection are implemented without chDB code or dependencies, with the relevant client tests passing.

Written by the indexing model from the issue text.