ClickHouse / ClickHouse/ClickHouse

Add a mergeJson SimpleAggregateFunction for AggregatingMergeTrees

Open
#77,094 0 comments 5 reactions 0 assignees View on GitHub
comp-aggregate-functions feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

[Langfuse](https://github.com/langfuse/langfuse)

### Use case

As part of our tables we have a `metadata` column which contains JSON style information which we currently store as `Map(LowCardinality(String), String)`. We're also open to move to the new JSON type if necessary for this feature.

Our users send us events for their tracing data, including changes to the metadata object. Those usually arrive in form of deltas, i.e. `{ foo: "bar" }` may arrive and later we get `{ bar: "baz"}`. The resulting object is meant to be `{ foo: "bar", bar: "baz" }`. Today, we hold the raw events in an object storage, read all raw events and then insert the merged row into a ReplacingMergeTree.

### Describe the solution you'd like

We want to insert the records as they come in into ClickHouse and use a Materialized View and an AggregatingMergeTree to perform this step within our warehouse as this removes the need to keep all raw versions stored indefinitely for when updates arrive. A SimpleAggrateFunction that merge incoming JSON objects would be extremely useful here. The semantics would essentially be the same as with [lodash merge](https://www.geeksforgeeks.org/lodash-_-merge-method/).

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.