WHERE on a struct field returned by RS_ZonalStatsAll fails: async functions should not be called directly
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
## Expected behavior
A `WHERE` predicate on a field of the struct returned by `RS_ZonalStatsAll` (or any other GDAL-backed, async raster function) should filter rows like any other predicate.
## Actual behavior
Any `WHERE` that references a field of the struct fails with:
```
sedonadb._lib.SedonaError: Internal error: async functions should not be called directly.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker
```
Aggregating over the same struct fields (`SUM(s.count)`, `GROUP BY name`) works. Filtering on a plain column next to the struct works. The failure is specific to a predicate that touches the async function's output; the planner appears to push the filter into the projection and evaluate the async UDF on the synchronous path.
Matrix (each query wraps the same inner join):
| shape | result |
|---|---|
| `SELECT SUM(s.count) FROM (inner)` | OK |
| `... GROUP BY name` | OK |
| `... WHERE s.count > 0` | FAIL |
| `... WHERE s.count > 0 GROUP BY name` | FAIL |
| CTE + `SUM` / `GROUP BY` | OK |
| CTE + `WHERE s.count > 0` | FAIL |
| `SELECT name, s.count FROM (inner) WHERE s.count > 0` (no aggregate) | FAIL |
| `... WHERE name <> 'x'` (predicate on a plain column) | OK |
| view over inner + `WHERE s.count > 0` | FAIL |
Workaround: materialize first (`df.to_memtable()`), then filter.
## Steps to reproduce
```python
import sedonadb
sd = sedonadb.connect()
# any GeoParquet with a geometry column; here 26 polygons
sd.read_parquet("cantons.parquet").to_view("cantons")
p = ("s3://copernicus-dem-90m/Copernicus_DSM_COG_30_N46_00_E006_00_DEM/"
"Copernicus_DSM_COG_30_N46_00_E006_00_DEM.tif") # public bucket, AWS_NO_SIGN_REQUEST=YES
sd.sql(f"SELECT RS_FromPath('{p}') AS rast").to_view("dem")
inner = """SELECT c.name, RS_ZonalStatsAll(d.rast, c.geometry) AS s
FROM dem d JOIN cantons c ON RS_Intersects(d.rast, c.geometry)"""
sd.sql(f"SELECT SUM(s.count) FROM ({inner})").to_pandas() # OK
sd.sql(f"SELECT SUM(s.count) FROM ({inner}) WHERE s.count > 0").to_pandas() # FAIL
sd.sql(f"SELECT name, s.count FROM ({inner}) WHERE s.count > 0").to_pandas() # FAIL
```
## Settings
SedonaDB version: 0.4.1 (PyPI wheel `sedonadb-0.4.1-cp311-cp311-macosx_12_0_arm64.whl`)
Python: 3.11
OS: macOS 15 (arm64)
GDAL: bundled with the wheel
Contributor guide
Research direction
Start by reproducing the failing RS_ZonalStatsAll query and compare it with the working aggregate and materialized-table cases. Trace the filter-planning path for predicates on async GDAL-backed raster UDF output; done means WHERE s.count works without to_memtable() and the listed query matrix passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100