ClickHouse / ClickHouse/ClickHouse
sessionization of events with PROJECTION feature
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Hi, I am testing an interesting usecase if PROJECTION feature. I wanted to hear your thoughts.
My end goal is to create a table that gets INSERTS from separate data streams.
The streams have different attributes about the session but all streams can be joined with a unique UUID session id.
The target table is an AggregateMergeTree that merges the different attributes into a single event by usage of a
`SimpleAggregateFunction(max, TYPE)`.
The table is ordered by several of the attributes of course for better compression and queries times.
This in turn creates a requirement to pass all the ORDER by fields to all the streams.
Now, about projections - I ran the below test. Its goal was to see if I can create a table with 1 merge key, and another composite ORDER BY key.
```SQL
CREATE TABLE `tests.projection_test`
(
`timestamp` DateTime Codec (DoubleDelta, LZ4),
`ne` Int32,
`request_id` String,
`a` SimpleAggregateFunction(max, String) ,
`b` SimpleAggregateFunction(max, Int32),
PROJECTION projection_1 (SELECT * ORDER BY ne)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toDate(timestamp)
ORDER BY (request_id)
SETTINGS index_granularity = 8192, storage_policy = 'ebs_hot_and_cold';
INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '1111-2222', '', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '1111-2222', 'a', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '0000-2222', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 10, '0000-2222', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 10, '1111-3333', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 9, '1111-4444', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 8, '1111-5555', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 8, '1111-5555', 'c', 1);
optimize table tests.projection_test final;
select * from tests.projection_test;
select * from tests.projection_test ORDER BY ne;
```
Observations:
1. It seems like this strategy works. `select * from tests.projection_test ORDER BY ne` does returns expected merged results. trace logs shows projection was used.
2. using ` WHERE timestamp >= ` uses projection by default ( IDK why that is, but in my use case it is indeed beneficial ).
3. As expected we doubled the table size.
To handle the doubling of the table size, my idea was to manipulate the projection. However I did not see any option that fits the need to drop the primary storage ( default table ), and use instead the projection.
What I did was deleting all files not necessary.
Before:

After:

This also worked.
1. `select * from tests.projection_test;` - FAILS
2. `select * from tests.projection_test ORDER BY ne;` - WORKS
3. `select * from tests.projection_test WHERE timestamp >= '2021-08-28'`; -WORKS
Questions:
1. general wdyt ?
2. Any specific issues you see ?
Contributor guide
Assessment
This issue has not been assessed yet.