Implement GDAL reading using sedona-gdal bindings
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
Currently we implement GDAL reading via OGR using pyogrio's Python interface. This leads to some Python calls that would be nice to avoid...among other things, when we have Python in multiple places in the plan (e.g., udfs, writes that use Python) we can get a hang. Even though we can probably fix those hangs otherwise, moving GDAL reads and/or writes into Rust means we can get them for free in SQL and R.
Roughly, we need to implement the Arrow interface here:
https://github.com/apache/sedona-db/blob/e3f57ff4714706a5839cce33537d0deccbcf6e7d/c/sedona-gdal/src/vector/layer.rs#L30-L35
...and wire up constructing the dataset and layer like pyogrio does. This is pretty involved, and handles the predicate pushdown (bbox, "ignored fields", limit).
https://github.com/geopandas/pyogrio/blob/main/pyogrio/_io.pyx#L1908-L1929
We currently implement the ExternalFormatSpec via Python...we just need to implement the ExternalFormatSpec directly in Rust.
Contributor guide
Research direction
Start with c/sedona-gdal/src/vector/layer.rs around the Arrow interface, then compare dataset and layer construction with pyogrio/_io.pyx lines 1908-1929. Trace the current Python ExternalFormatSpec path and identify how bbox, ignored fields, and limit predicate pushdown are handled. Done means GDAL reading constructs the dataset and layer and implements ExternalFormatSpec directly in Rust.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100