ClickHouse / ClickHouse/ClickHouse

Reduce high memory usage in advanced shared data serialization with huge number of unque and sparse paths, especially for JSON inside Arrays

Open
#108,992 3 comments 1 reaction 1 assignee Claimed by @Avogar View on GitHub
comp-json-datatype memory
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Problem

During merge, `flattenAndBucketSharedDataPaths` in the ADVANCED shared data serialization materializes all shared data paths into separate in-memory `ColumnDynamic` columns simultaneously across all buckets for the entire row range at once. When a JSON column has many unique paths in SharedData, this leads to excessive memory usage proportional to `num_unique_paths * num_rows * 9 bytes` (1 byte NULL discriminator + 8 bytes offset per default row per path).

This is especially problematic when JSON is nested inside an Array (`Array(JSON)`), because the row count passed to `flattenAndBucketSharedDataPaths` is the total number of array elements (which can be orders of magnitude larger than the outer row count). For top-level JSON columns the row range is bounded by `index_granularity`, but for nested arrays there is no such limit.

### Proposed fixes

#### 1. Per-bucket extraction

Process one bucket at a time instead of materializing all buckets simultaneously.

Currently `flattenAndBucketSharedDataPaths` iterates shared data once and creates `ColumnDynamic` objects for **all** paths across **all** buckets. Instead, iterate shared data once per bucket, creating `ColumnDynamic` objects only for paths belonging to the current bucket. After serializing each bucket's data, free its columns before processing the next bucket.

- **Memory reduction**: ~Nx (where N = number of buckets)
- **CPU cost**: Same total work. The `insertDefault` loop cost is `paths_per_bucket * total_rows` per bucket, summing to `total_paths * total_rows` — identical to the current approach. Shared data iteration overhead (N hash computations per entry) is negligible.
- **Format change**: None. Serialized output is identical.

#### 2. Row chunk limit

Add a configurable limit on the number of rows processed per invocation of `flattenAndBucketSharedDataPaths`. When the row range `(end - start)` exceeds this limit, split into multiple chunks, each processed and serialized independently. This is analogous to how `index_granularity` limits the row count per granule for top-level columns, but applied to the nested array element level.

- **Memory reduction**: Proportional to `total_rows / chunk_size`. Each chunk also naturally limits the number of paths seen (only paths present in those rows are materialized).
- **CPU cost**: Same total work for the `insertDefault` loop.
- **Format change**: None. The current serialization already processes data granule by granule and can produce multiple chunks per granule, so splitting into smaller row chunks fits naturally into the existing format.
- **Combined with Fix 1**: `(total_paths / N)` paths/bucket * `chunk_size` rows/chunk * 9 bytes per chunk per bucket.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.