ClickHouse / ClickHouse/ClickHouse
Vertical insert (batched column writes) for very wide MergeTree tables?
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Use case
I have very wide MergeTree tables (~10k columns).
Even modest inserts (e.g., ~1k rows) can spike memory usage to 16+ GB because the current insert path keeps all columns in memory while sorting.
I need a lower‑memory insert path for wide tables.
### Describe the solution you'd like
A “vertical insert” mode modeled after vertical merge: write key/sorting columns first (horizontal phase to establish ordering + index granularity), then write remaining columns in batches using the same
permutation.
Batching is important — I tried one‑column‑at‑a‑time and the throughput hit was too severe.
Ideally this is gated by settings/thresholds and limited to Wide parts.
### Describe alternatives you've considered
- One‑column‑at‑a‑time vertical inserts: too slow in practice.
- Increasing memory limits: not feasible for large concurrent loads.
- Manual column sharding or splitting tables: adds complexity and breaks existing schema expectations.
- Pre‑sorting or splitting inserts: presort does not help much. Splitting inserts helps, but even small number of rows (~1000) balloon in memory usage (~16GB).
### Additional context
I’m happy to implement the feature and iterate based on maintainers’ guidance.
I’m mostly looking for feedback on whether this direction aligns with ClickHouse’s design goals and any pitfalls around indexes/TTL/projections/merge variants.
Contributor guide
Assessment
This issue has not been assessed yet.