apache / apache/datafusion-python

Simplified filter expression has a Null type instead of Int64 type across the FFI layer

未關閉
#1,551 11 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
604
分支
174
平均合併
1 天 7 小時
30 天內合併 PR
4

描述

**Describe the bug**

A custom table provider for a ParquetSource with a trivial catalog and an Int64 column yields some errors when a SQL query has a filter with a literal limit on that column of the form

```
assertion `left == right` failed: Simplified expression should have the same data type as the original
left: Null
right: Int64
```

The error does not occur when using datafusion-python 52; it also does not occur when running the query purely in a Rust SessionContext; the backtrace for the above error shows it coming from datafusion-ffi code as well.

This may of course not be a bug, but instead some bad practice that the version 52 set of crates tolerates but which is now invalid. A MRE of this custom table provider can be found in the public repo https://github.com/jwimberl/datafusion_python_53_int64filter_repro, which contains

- a non-working datafusion53 version (in branch [main](https://github.com/jwimberl/datafusion_python_53_int64filter_repro))
- a baseline working version (in branch [datafusion52](https://github.com/jwimberl/datafusion_python_53_int64filter_repro/tree/datafusion52))

and a canned dummy dataset. The README.md of this repo has more details.

**To Reproduce**

In the `main` branch, build the `py_repro_provider` crate with `maturin develop` and run `python repro.py`. This loads the dummy dataset as a table `dummy_table` and runs two queries
- `SELECT * FROM dummy_table LIMIT 1`, which is successful
- `SELECT * FROM dummy_table LIMIT 5`, which panics

Itss output should be something like

```
Successful query:
a b
0 0 42
Unsuccesful query:

thread '' panicked at /home/jwimberley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-expr-53.1.0/src/simplifier/mod.rs:76:17:
assertion `left == right` failed: Simplified expression should have the same data type as the original
left: Null
right: Int64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

followed by backtrace information.

**Expected behavior**

In the `datafusion52` branch, build the`py_repro_provider` with `maturin develop` and run `python repro.py`. It runs the same two queries, and its output should be

```
Successful query:
a b
0 0 42
Also successful query:
a b
0 0 42
1 1 42
2 2 42
3 3 42
4 4 42
```

**Additional context**

In either the `main` branch or `datafusion52` branch, the Rust code for the table provider is in the directory `repro_provider`, and there is a corresponding cargo test that runs `SELECT * FROM dummy_table WHERE a < 5`. Without the FFI layer, this is successful with both datafusion 52 and 53.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

先從連結的 MRE 開始,尤其是 repro.py 和 repro_provider 目錄,接著比較 main 和 datafusion52 分支。執行 maturin develop 和 python repro.py 步驟以重現 LIMIT 5 panic,並執行針對 SELECT * FROM dummy_table WHERE a < 5 的對應 cargo test,以比較有無 FFI 層時的行為。完成的標準是 FFI 查詢能夠成功執行,且不出現 Null-versus-Int64 assertion。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python, rust
領域
backend, data-engineering
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。