developmentseed / developmentseed/cql2-rs
[Discussion] OGC Features query to SQL
- Dominant language
- Rust
- Stars
- 23
- Forks
- 3
- Avg merge
- 5h 2m
- Merged PRs (30d)
- 7
Description
I recently worked on [converting tipg to use cql2-rs](https://github.com/developmentseed/tipg/pull/260) and it occurred to me that I might be able to convert the query string parsing to a more "standalone" function. It would take in all query parameters (not just the `filter`) and return a full SQL `WHERE` statement.
This is something I can take on, but in order to maximize usefulness I'd like to **openly discuss** this idea. I'm totally open to thoughts/feedback. Possible points of discussion:
- Why?
- Which project should this live in?
- Which parts of OGC Features could be reasonable implemented?
- Could this be made reasonably backend independant?
- What would be the minimum goals/acceptance criteria?
## My Answers
**Why?**
- When I first came across this project, I initially thought that was exactly what `to_sql()` did 😅 so _imagine my shock_ when I had to do more work! but more seriously, if I was surprised by this, others might be too.
- Implementing this would also generalize the "spec compliance" away from other Features API implementations or backends. `tipg` or something like it could focus on the surrounding code, and get an automatic boost towards full compliance.
- A utility like this could be used outside of `tipg` or any one Features API implementation.
- Implementing other parts of the Features API along with the filter (parsed by `cql2-rs`) requires a fair amount of interweaving with `cql2-rs` (see [implementation of `cql_where()` in tipg](https://github.com/developmentseed/tipg/blob/main/tipg/collections.py#L428)). In fact, the treatment of `bbox`, `properties`, `datetime`, etc _are_ just additional cql2 clauses. So while it's a nice separation of concern to say "`cql2-rs` is responsible for parsing of the `filter` parameter only" in practice, the rest of the query parameters are already tightly linked to cql2, the `filter` parameter, and `cql2-rs`.
**Which project?**
There's 3 reasonable places to put something like this:
- **In `cql2-rs`**. As mentioned, an implementation of this will end up being tightly coupled to `cql2-rs`. It could also be argued that this is just a "helper" function that wraps existing `.to_sql()` already here. It could be added with no additional dependencies.
- **In `tipg`**. An obvious choice, but my initial use case is actually to use this _without_ tipg and independently of an actual full-fledged Features API (I would be using this to generate new `VIEW` statements). Adding it to `tipg` might also bind it closer to a single database or Features API implementation.
- **A new project**. Being just a "helper" to `cql2-rs`, and frankly not being that big, it seems a little unnecessary to be a completely independent project with all the overhead that comes with that. It would also require `tipg` to have an additional dependency and the required version tracking.
Personally, I would lean towards putting it in here, with a HUGE caveat: I'm not a Rust programmer, so I would have to implement it in Python.
**Which parts of OGC Features could be reasonable implemented?**
I think as a first pass I would basically rip out the current implementation from `tipg`. So notably no Part 2 (CRS by Reference).
I do **not** think this should construct the whole "tile" query `ST_AsMVT`. Tile serving in a performant way is hard, and this is better left up to a specific API.
**Could this be made reasonably backend independant?**
Yes (I say with 90% confidence). I believe all the needed SQL statements including `ST_` functions needed are the same between say Postgres and DuckDB. Definitely more thought needed, and if there's a glaring reason why this could not be made DB independant, please say!
**What would be the minimum goals/acceptance criteria?**
I think the initial goal would be to replace a big chunk of code in `tipg`, focusing on the query parameters that are actually part of OGC Features. Followed by adding other parameters and then ensuring DB independence.
I think it's worth putting in writing that this would do no DB-side validation. This script (as `cql2-rs` currently does) would take `properties` and other parameters and assume that they are valid column names. This would do no DB connections.
---
Those are just my thoughts for now. Open to any feedback or discussion around this. Thanks!
cc @vincentsarago @bitner as you might be closest to these 2 projects.
Contributor guide
Research direction
Start by reading the existing .to_sql() implementation in cql2-rs and tipg's cql_where() in tipg/collections.py. Compare how OGC Features query parameters such as filter, bbox, properties, and datetime are handled. Done means agreed acceptance criteria and a backend-independent standalone SQL WHERE generator, without DB connections or tile-query construction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100