Does bloom filter in SQL even work?
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
The docs at https://druid.apache.org/docs/latest/development/extensions-core/bloom-filter/ say the following:
> Bloom filters can be computed in SQL expressions with the bloom_filter aggregator:
> SELECT BLOOM_FILTER(, ) FROM druid.foo WHERE dim2 = 'abc'
> but requires the setting druid.sql.planner.serializeComplexValues to be set to true. Bloom filter results in a SQL response are serialized into a base64 string, which can then be used in subsequent queries as a filter.
I'm trying to do exactly that using the default `kttm_rollup` dataset, e.g.:
```
WITH yunowork AS (
SELECT
BLOOM_FILTER("ip_address", 10000) AS bf
FROM druid.kttm_rollup
)
SELECT
ip_address
FROM druid.kttm_rollup
JOIN yunowork Y ON 1 = 1
WHERE BLOOM_FILTER_TEST(ip_address, Y.bf)
```
I get the following error:
```
Error: INVALID_INPUT
Cannot apply 'BLOOM_FILTER_TEST' to arguments of type 'BLOOM_FILTER_TEST(, >)'. Supported form(s): 'BLOOM_FILTER_TEST(, )' (line [10], column [7])
```
`druid.sql.planner.serializeComplexValues` is set to true, but as far as I understand [it's irrelevant](https://github.com/apache/druid/pull/17549).
Am I doing something wrong?
### Affected Version
31.0.1 (locally on osx) and 26.0.0 (cluster on linux), I assume all versions in between too.
### Description
Mostly described above.
Contributor guide
Research direction
Start with the bloom-filter documentation and reproduce the provided query against the default kttm_rollup dataset. Investigate the BLOOM_FILTER and BLOOM_FILTER_TEST SQL entry points, including the serializeComplexValues setting and the reported argument-type error. Done means the documented workflow works or the documentation accurately explains the supported usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100