ClickHouse / ClickHouse/ClickHouse
how to use roaringbitmap filter with normal table efficient.
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
I split the roaringbitmap and write it to the clickhouse local table。but i can't get correct result in efficient.
### Question
table1:
dwd_agg_bitmap_table (
`pt_d` Date,
`tag_id` UInt32,
`tag_bitmap` AggregateFunction(groupBitmap, UInt64)
)
datas follow:
SELECT bitmapToArray(tag_bitmap) FROM dwd_agg_bitmap_table_**local** WHERE tag_id ='100001';
**node1:**
[0,2,4,6,8,10]
**node2:**
[1,3,5,7,9]
table2:
dwd_normal_table (
`pt_d` Date DEFAULT now() COMMENT 'pt_d',
`did_bigint` UInt64 COMMENT 'did_bigint'
)
datas follow:
SELECT groupArray(did_bigint) FROM dwd_normal_table_**local** where toYYYYMMDD(pt_d) = 20240723;
**node1:**
[0,2]
**node2:**
[1,3]
I want to get the intersection of two tables. so i try follows sql:
**try1:**
SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT tag_bitmap FROM dwd_agg_bitmap_table_**all** WHERE tag_id ='100001'),did_bigint);
but any node it's bad: DB::Exception: Scalar subquery returned more than one row: While processing
**try2:**
SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT tag_bitmap FROM dwd_agg_bitmap_table_**local** WHERE tag_id ='100001'),did_bigint);
sql execute ok. i hopes should give me [0,1,2,3] . but result it's not my expected.
node1 execute:
[0,2]
node2 execute:
[1,3]
**try3:**
i just use follow sql. it‘s’ expected correct in any node
SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT **groupBitmapMergeState**(tag_bitmap) FROM dwd_agg_bitmap_table_all WHERE tag_id ='100001'),did_bigint);
node1 execute:
[0,2,1,3]
node2 execute:
[0,2,1,3]
i think use **groupBitmapMergeState** Will aggregate bitmap into 1, it's not my expected , i split bitmap to more in all node. hopes like :
node1: normal_table_local bitmapCoantains(bitmap_table_local) get result_node1
node2: normal_table_local bitmapCoantains(bitmap_table_local) get result_node2
distributed result give me result1_node1+result1_node2 .
so how can i impl it. use one sql get correct result like the steps above.
Seek help from experienced any one !
Contributor guide
Assessment
This issue has not been assessed yet.