duckdb / duckdb/duckdb-python

Arrow scan: a NULL row passes a pushed-down `<=` filter against a NaN constant

Abierto
#615 0 comentarios 0 reacciones 0 asignados Ver en GitHub
needs triage
Lenguaje dominante
Python
Estrellas
186
Forks
113
Merge medio
13 h 29 min
PR fusionados (30 d)
17

Descripción

### What happens?

When a `WHERE b <= 'nan'::DOUBLE` filter is pushed into an Arrow scan, the scan returns rows where b is NULL.

The same predicate over same rows copied to a native table (correctly) excludes the null rows.

This *should* exclude the nulls (`NULL <= x` is false): The bug appears to be in the arrow filter_pushdown: disabling the optimizer/pushdowns returns the correct results.

### To Reproduce

```py
import duckdb
import pyarrow as pa

with duckdb.connect() as con:

print(f"duckdb {duckdb.__version__}, pyarrow {pa.__version__}", "source id:", con.execute("SELECT source_id FROM pragma_version()").fetchone()[0])

arrow_table = pa.table({"b": pa.array([1.0, float("nan"), None])})
con.register("arrow_t", arrow_table)

result_native = con.execute("create table native_t as select * from arrow_t;select b from native_t where b <= 'nan'::DOUBLE").fetchall()

result = con.execute("select b from arrow_t where b <= 'nan'::DOUBLE").fetchall()

con.execute("PRAGMA disable_optimizer;")
result_no_optimizer = con.execute("select b from arrow_t where b <= 'nan'::DOUBLE").fetchall()

print(f"{result_native=}")
print(f"{result=}")
print(f"{result_no_optimizer=}")
```

## Result: Note that "result" differs from others
>
> duckdb 1.6.0.dev379, pyarrow 25.0.1 source id: a00803f768
> result_native=[(1.0,), (nan,)]
> result=[(1.0,), (nan,), (None,)]
> result_no_optimizer=[(1.0,), (nan,)]

### OS:

Windows & Ubuntu WSL

### DuckDB Package Version:

1.5.5 and 1.6.0.dev379

### Python Version:

3.14

### Full Name:

Paul T

### Affiliation:

Iqmo

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

### Did you include all relevant data sets for reproducing the issue?

Yes

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration to reproduce the issue?

- [x] Yes, I have

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con la reproducción en Python usando la tabla PyArrow registrada y compara el escaneo de Arrow con los resultados de native-table y optimizer-disabled. Traza el filter pushdown de Arrow para `b <= 'nan'::DOUBLE`; se considera terminado cuando la consulta con pushdown excluye la fila NULL, al igual que los otros resultados, y una prueba de regresión cubre el caso.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
databases
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
68/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.