ClickHouse / ClickHouse/ClickHouse
Add a mergeJson SimpleAggregateFunction for AggregatingMergeTrees
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
[Langfuse](https://github.com/langfuse/langfuse)
### Use case
As part of our tables we have a `metadata` column which contains JSON style information which we currently store as `Map(LowCardinality(String), String)`. We're also open to move to the new JSON type if necessary for this feature.
Our users send us events for their tracing data, including changes to the metadata object. Those usually arrive in form of deltas, i.e. `{ foo: "bar" }` may arrive and later we get `{ bar: "baz"}`. The resulting object is meant to be `{ foo: "bar", bar: "baz" }`. Today, we hold the raw events in an object storage, read all raw events and then insert the merged row into a ReplacingMergeTree.
### Describe the solution you'd like
We want to insert the records as they come in into ClickHouse and use a Materialized View and an AggregatingMergeTree to perform this step within our warehouse as this removes the need to keep all raw versions stored indefinitely for when updates arrive. A SimpleAggrateFunction that merge incoming JSON objects would be extremely useful here. The semantics would essentially be the same as with [lodash merge](https://www.geeksforgeeks.org/lodash-_-merge-method/).
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.