duckdb / duckdb/duckdb-python

Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans

Open
#463 3 comments 0 reactions 0 assignees View on GitHub
needs more info
Dominant language
Python
Stars
186
Forks
113
Avg merge
13h 29m
Merged PRs (30d)
17

Description

### What happens?

Hi,

I am running into this error message and am unsure what to do.
```
_duckdb.NotImplementedException: Not implemented Error: Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans
```
It seems I can either disable pushdown filtering through joins, or materialize the query before doing my join. I would rather not do either, since it is a lot of data that I am processing.

Reading the source files with a non-pyarrow reader is unfortunately not an option at this time.

### To Reproduce

Unfortunately, I don't really have a reproducer, but I think the maintainers will know what the issue is anyways from the error message. It seems to me that Pyarrow do support bloom filter, but the pushdown is just not implemented?

### OS:

Ubuntu 22

### DuckDB Package Version:

1.5.2

### Python Version:

3.11

### Full Name:

Jakob Stigenberg

### Affiliation:

Qubos Systematic

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

### Did you include all code required to reproduce the issue?

- [ ] Yes, I have

### Did you include all relevant configuration to reproduce the issue?

- [x] Yes, I have

Contributor guide

Open the contributing guide

Research direction

Start at the PyArrow scan path that emits the BLOOM_FILTER NotImplementedException and investigate how join filter pushdown reaches it. The report has no reproducible dataset or code, so first establish a minimal case using DuckDB 1.5.2 with Python 3.11. Done means the supported PyArrow scan no longer raises this error for the reported pushdown scenario, with behavior validated against an appropriate regression case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.