ClickHouse / ClickHouse/ClickHouse
Reduce high memory usage in advanced shared data serialization with huge number of unque and sparse paths, especially for JSON inside Arrays
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Problem
During merge, `flattenAndBucketSharedDataPaths` in the ADVANCED shared data serialization materializes all shared data paths into separate in-memory `ColumnDynamic` columns simultaneously across all buckets for the entire row range at once. When a JSON column has many unique paths in SharedData, this leads to excessive memory usage proportional to `num_unique_paths * num_rows * 9 bytes` (1 byte NULL discriminator + 8 bytes offset per default row per path).
This is especially problematic when JSON is nested inside an Array (`Array(JSON)`), because the row count passed to `flattenAndBucketSharedDataPaths` is the total number of array elements (which can be orders of magnitude larger than the outer row count). For top-level JSON columns the row range is bounded by `index_granularity`, but for nested arrays there is no such limit.
### Proposed fixes
#### 1. Per-bucket extraction
Process one bucket at a time instead of materializing all buckets simultaneously.
Currently `flattenAndBucketSharedDataPaths` iterates shared data once and creates `ColumnDynamic` objects for **all** paths across **all** buckets. Instead, iterate shared data once per bucket, creating `ColumnDynamic` objects only for paths belonging to the current bucket. After serializing each bucket's data, free its columns before processing the next bucket.
- **Memory reduction**: ~Nx (where N = number of buckets)
- **CPU cost**: Same total work. The `insertDefault` loop cost is `paths_per_bucket * total_rows` per bucket, summing to `total_paths * total_rows` — identical to the current approach. Shared data iteration overhead (N hash computations per entry) is negligible.
- **Format change**: None. Serialized output is identical.
#### 2. Row chunk limit
Add a configurable limit on the number of rows processed per invocation of `flattenAndBucketSharedDataPaths`. When the row range `(end - start)` exceeds this limit, split into multiple chunks, each processed and serialized independently. This is analogous to how `index_granularity` limits the row count per granule for top-level columns, but applied to the nested array element level.
- **Memory reduction**: Proportional to `total_rows / chunk_size`. Each chunk also naturally limits the number of paths seen (only paths present in those rows are materialized).
- **CPU cost**: Same total work for the `insertDefault` loop.
- **Format change**: None. The current serialization already processes data granule by granule and can produce multiple chunks per granule, so splitting into smaller row chunks fits naturally into the existing format.
- **Combined with Fix 1**: `(total_paths / N)` paths/bucket * `chunk_size` rows/chunk * 9 bytes per chunk per bucket.
Contributor guide
Assessment
This issue has not been assessed yet.