apache / apache/druid

Does bloom filter in SQL even work?

Open
#17,602 3 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

The docs at https://druid.apache.org/docs/latest/development/extensions-core/bloom-filter/ say the following:

> Bloom filters can be computed in SQL expressions with the bloom_filter aggregator:
> SELECT BLOOM_FILTER(, ) FROM druid.foo WHERE dim2 = 'abc'
> but requires the setting druid.sql.planner.serializeComplexValues to be set to true. Bloom filter results in a SQL response are serialized into a base64 string, which can then be used in subsequent queries as a filter.

I'm trying to do exactly that using the default `kttm_rollup` dataset, e.g.:

```
WITH yunowork AS (
SELECT
BLOOM_FILTER("ip_address", 10000) AS bf
FROM druid.kttm_rollup
)
SELECT
ip_address
FROM druid.kttm_rollup
JOIN yunowork Y ON 1 = 1
WHERE BLOOM_FILTER_TEST(ip_address, Y.bf)
```

I get the following error:

```
Error: INVALID_INPUT

Cannot apply 'BLOOM_FILTER_TEST' to arguments of type 'BLOOM_FILTER_TEST(, >)'. Supported form(s): 'BLOOM_FILTER_TEST(, )' (line [10], column [7])
```

`druid.sql.planner.serializeComplexValues` is set to true, but as far as I understand [it's irrelevant](https://github.com/apache/druid/pull/17549).

Am I doing something wrong?

### Affected Version

31.0.1 (locally on osx) and 26.0.0 (cluster on linux), I assume all versions in between too.

### Description

Mostly described above.

Contributor guide

Open the contributing guide

Research direction

Start with the bloom-filter documentation and reproduce the provided query against the default kttm_rollup dataset. Investigate the BLOOM_FILTER and BLOOM_FILTER_TEST SQL entry points, including the serializeComplexValues setting and the reported argument-type error. Done means the documented workflow works or the documentation accurately explains the supported usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.