ClickHouse / ClickHouse/ClickHouse

Vertical insert (batched column writes) for very wide MergeTree tables?

Open
#93,366 9 comments 0 reactions 0 assignees View on GitHub
comp-mergetree external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Use case

I have very wide MergeTree tables (~10k columns).

Even modest inserts (e.g., ~1k rows) can spike memory usage to 16+ GB because the current insert path keeps all columns in memory while sorting.

I need a lower‑memory insert path for wide tables.

### Describe the solution you'd like

A “vertical insert” mode modeled after vertical merge: write key/sorting columns first (horizontal phase to establish ordering + index granularity), then write remaining columns in batches using the same
permutation.

Batching is important — I tried one‑column‑at‑a‑time and the throughput hit was too severe.
Ideally this is gated by settings/thresholds and limited to Wide parts.

### Describe alternatives you've considered

- One‑column‑at‑a‑time vertical inserts: too slow in practice.
- Increasing memory limits: not feasible for large concurrent loads.
- Manual column sharding or splitting tables: adds complexity and breaks existing schema expectations.
- Pre‑sorting or splitting inserts: presort does not help much. Splitting inserts helps, but even small number of rows (~1000) balloon in memory usage (~16GB).

### Additional context

I’m happy to implement the feature and iterate based on maintainers’ guidance.

I’m mostly looking for feedback on whether this direction aligns with ClickHouse’s design goals and any pitfalls around indexes/TTL/projections/merge variants.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.