ClickHouse / ClickHouse/ClickHouse

sessionization of events with PROJECTION feature

Open
#28,311 12 comments 0 reactions 0 assignees View on GitHub
comp-projections external question
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

Hi, I am testing an interesting usecase if PROJECTION feature. I wanted to hear your thoughts.

My end goal is to create a table that gets INSERTS from separate data streams.

The streams have different attributes about the session but all streams can be joined with a unique UUID session id.

The target table is an AggregateMergeTree that merges the different attributes into a single event by usage of a
`SimpleAggregateFunction(max, TYPE)`.

The table is ordered by several of the attributes of course for better compression and queries times.
This in turn creates a requirement to pass all the ORDER by fields to all the streams.

Now, about projections - I ran the below test. Its goal was to see if I can create a table with 1 merge key, and another composite ORDER BY key.

```SQL
CREATE TABLE `tests.projection_test`
(
`timestamp` DateTime Codec (DoubleDelta, LZ4),
`ne` Int32,

`request_id` String,
`a` SimpleAggregateFunction(max, String) ,
`b` SimpleAggregateFunction(max, Int32),

PROJECTION projection_1 (SELECT * ORDER BY ne)

)
ENGINE = AggregatingMergeTree()
PARTITION BY toDate(timestamp)
ORDER BY (request_id)
SETTINGS index_granularity = 8192, storage_policy = 'ebs_hot_and_cold';

INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '1111-2222', '', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '1111-2222', 'a', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 0, '0000-2222', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 10, '0000-2222', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 10, '1111-3333', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 9, '1111-4444', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 8, '1111-5555', 'b', 0);
INSERT INTO tests.projection_test VALUES ('2021-08-29', 8, '1111-5555', 'c', 1);

optimize table tests.projection_test final;
select * from tests.projection_test;
select * from tests.projection_test ORDER BY ne;
```

Observations:
1. It seems like this strategy works. `select * from tests.projection_test ORDER BY ne` does returns expected merged results. trace logs shows projection was used.
2. using ` WHERE timestamp >= ` uses projection by default ( IDK why that is, but in my use case it is indeed beneficial ).
3. As expected we doubled the table size.

To handle the doubling of the table size, my idea was to manipulate the projection. However I did not see any option that fits the need to drop the primary storage ( default table ), and use instead the projection.
What I did was deleting all files not necessary.

Before:

![image](https://user-images.githubusercontent.com/12124039/131248074-cda8c97d-91fc-4e63-a676-e4d4820cc311.png)

After:

![image](https://user-images.githubusercontent.com/12124039/131248100-786ec9d5-dce4-41b3-8f6e-2b9616a21afb.png)

This also worked.
1. `select * from tests.projection_test;` - FAILS
2. `select * from tests.projection_test ORDER BY ne;` - WORKS
3. `select * from tests.projection_test WHERE timestamp >= '2021-08-28'`; -WORKS

Questions:
1. general wdyt ?
2. Any specific issues you see ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.