ClickHouse / ClickHouse/ClickHouse

how to use roaringbitmap filter with normal table efficient.

Open
#68,491 0 comments 0 reactions 0 assignees View on GitHub
comp-distributed external question
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

I split the roaringbitmap and write it to the clickhouse local table。but i can't get correct result in efficient.

### Question

table1:
dwd_agg_bitmap_table (
`pt_d` Date,
`tag_id` UInt32,
`tag_bitmap` AggregateFunction(groupBitmap, UInt64)
)
datas follow:
SELECT bitmapToArray(tag_bitmap) FROM dwd_agg_bitmap_table_**local** WHERE tag_id ='100001';
**node1:**
[0,2,4,6,8,10]
**node2:**
[1,3,5,7,9]

table2:
dwd_normal_table (
`pt_d` Date DEFAULT now() COMMENT 'pt_d',
`did_bigint` UInt64 COMMENT 'did_bigint'
)
datas follow:
SELECT groupArray(did_bigint) FROM dwd_normal_table_**local** where toYYYYMMDD(pt_d) = 20240723;
**node1:**
[0,2]
**node2:**
[1,3]

I want to get the intersection of two tables. so i try follows sql:

**try1:**
SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT tag_bitmap FROM dwd_agg_bitmap_table_**all** WHERE tag_id ='100001'),did_bigint);

but any node it's bad: DB::Exception: Scalar subquery returned more than one row: While processing

**try2:**
SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT tag_bitmap FROM dwd_agg_bitmap_table_**local** WHERE tag_id ='100001'),did_bigint);
sql execute ok. i hopes should give me [0,1,2,3] . but result it's not my expected.
node1 execute:
[0,2]
node2 execute:
[1,3]

**try3:**
i just use follow sql. it‘s’ expected correct in any node

SELECT groupArray(did_bigint) FROM dwd_normal_table_**all** where bitmapContains((SELECT **groupBitmapMergeState**(tag_bitmap) FROM dwd_agg_bitmap_table_all WHERE tag_id ='100001'),did_bigint);

node1 execute:
[0,2,1,3]
node2 execute:
[0,2,1,3]

i think use **groupBitmapMergeState** Will aggregate bitmap into 1, it's not my expected , i split bitmap to more in all node. hopes like :
node1: normal_table_local bitmapCoantains(bitmap_table_local) get result_node1
node2: normal_table_local bitmapCoantains(bitmap_table_local) get result_node2
distributed result give me result1_node1+result1_node2 .

so how can i impl it. use one sql get correct result like the steps above.

Seek help from experienced any one !

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.