apache / apache/sedona-db

Implement GDAL reading using sedona-gdal bindings

Open
#1,092 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

Currently we implement GDAL reading via OGR using pyogrio's Python interface. This leads to some Python calls that would be nice to avoid...among other things, when we have Python in multiple places in the plan (e.g., udfs, writes that use Python) we can get a hang. Even though we can probably fix those hangs otherwise, moving GDAL reads and/or writes into Rust means we can get them for free in SQL and R.

Roughly, we need to implement the Arrow interface here:

https://github.com/apache/sedona-db/blob/e3f57ff4714706a5839cce33537d0deccbcf6e7d/c/sedona-gdal/src/vector/layer.rs#L30-L35

...and wire up constructing the dataset and layer like pyogrio does. This is pretty involved, and handles the predicate pushdown (bbox, "ignored fields", limit).

https://github.com/geopandas/pyogrio/blob/main/pyogrio/_io.pyx#L1908-L1929

We currently implement the ExternalFormatSpec via Python...we just need to implement the ExternalFormatSpec directly in Rust.

Contributor guide

Open the contributing guide

Research direction

Start with c/sedona-gdal/src/vector/layer.rs around the Arrow interface, then compare dataset and layer construction with pyogrio/_io.pyx lines 1908-1929. Trace the current Python ExternalFormatSpec path and identify how bbox, ignored fields, and limit predicate pushdown are handled. Done means GDAL reading constructs the dataset and layer and implements ExternalFormatSpec directly in Rust.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.