ClickHouse / ClickHouse/clickhouse-js

RowBinary: dynamic (runtime-schema) writer — do you need one?

Open
#919 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
331
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Background

The RowBinary codec has a **dynamic reader**: `compileRowBinaryWithNamesAndTypes` reads the `RowBinaryWithNamesAndTypes` header, folds each column's type string into a `Reader` (`astToReader`), and decodes the rest of the stream — a generic way to consume *any* response without knowing the schema ahead of time.

There is no equivalent **dynamic writer** today, and that's intentional. This issue exists mostly to explain *why*, and to help anyone who lands here decide whether they actually need one.

## You probably don't need this

The read and write sides aren't symmetric in practice:

- When **reading** `RowBinaryWithNamesAndTypes`, the **server** decides the schema and sends it to you in the header — the projection, the types after functions/aggregations, the column order. The client genuinely can't know them ahead of time, so a runtime-driven decoder is the natural fit.
- When **writing**, you're usually inserting *your* data into *your* table, whose schema you already know at build time. In that case the specialized, hand-written writer (or a generated one) is the better choice — it's faster and simpler, and the codec is designed to steer you there.

So if you're encoding application data into a known table, you most likely want the specialized writer path described in [`writer.md`](../blob/main/skills/clickhouse-js-node-rowbinary/writer.md), not a dynamic one.

## When a dynamic writer might genuinely help

A runtime-schema-driven encoder mainly makes sense when the **code is decoupled from the schema it's encoding** — i.e. generic infrastructure rather than application code. A few scenarios:

1. **Generic ingestion tools / connectors** — a Kafka sink, ETL/CDC pipeline, or a "load this file into that table" utility that inserts into tables it doesn't know at build time. It would learn the schema at runtime (e.g. from `DESCRIBE TABLE` / `system.columns`) and encode rows accordingly. This is the closest mirror to the dynamic reader; the only difference is the schema comes from a query rather than a response header.
2. **`RowBinaryWithNamesAndTypes` as an *insert* format** — ClickHouse can accept that format as input and match columns to the target table by name. Emitting the names+types header + body for an unknown or evolving table benefits from building the encoders at runtime.
3. **Read-transform-write / copy** — table-to-table copy, fan-out, backfills, or an anonymizing proxy that decodes one stream and re-encodes it elsewhere, ideally driven by the same parsed type strings on both sides.
4. **Multi-tenant or evolving schemas** — one binary serving many slightly-different (or changing) tables, discovering and caching encoders per schema at runtime instead of being recompiled per schema.

## What it could look like

The write path would mirror the read path: an `astToWriter` (the inverse of `astToReader`) that folds a parsed [`@clickhouse/datatype-parser`](../tree/main/packages/datatype-parser) AST into a `Writer` using the existing generic combinators, plus a helper that emits the names+types header and returns a `writeRows` driver. As with the dynamic reader, it would be generic and unoptimized — the specialized path stays the recommendation whenever the types are fixed.

## Status

Not currently planned. If one of the scenarios above describes your use case, a comment with the details would help gauge interest — especially the generic-ingestion case, which is the strongest mirror of the reader.

Contributor guide

Open the contributing guide

Research direction

Read writer.md and the dynamic reader entry point compileRowBinaryWithNamesAndTypes, then compare astToReader with the existing writer combinators. The issue is not currently planned, so first validate a concrete runtime-schema use case; completion would require an agreed design for astToWriter and the names-and-types writeRows driver.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.