developmentseed / developmentseed/cql2-rs

[Discussion] OGC Features query to SQL

Open
#235 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
23
Forks
3
Avg merge
5h 2m
Merged PRs (30d)
7

Description

I recently worked on [converting tipg to use cql2-rs](https://github.com/developmentseed/tipg/pull/260) and it occurred to me that I might be able to convert the query string parsing to a more "standalone" function. It would take in all query parameters (not just the `filter`) and return a full SQL `WHERE` statement.

This is something I can take on, but in order to maximize usefulness I'd like to **openly discuss** this idea. I'm totally open to thoughts/feedback. Possible points of discussion:

- Why?
- Which project should this live in?
- Which parts of OGC Features could be reasonable implemented?
- Could this be made reasonably backend independant?
- What would be the minimum goals/acceptance criteria?

## My Answers
**Why?**
- When I first came across this project, I initially thought that was exactly what `to_sql()` did 😅 so _imagine my shock_ when I had to do more work! but more seriously, if I was surprised by this, others might be too.
- Implementing this would also generalize the "spec compliance" away from other Features API implementations or backends. `tipg` or something like it could focus on the surrounding code, and get an automatic boost towards full compliance.
- A utility like this could be used outside of `tipg` or any one Features API implementation.
- Implementing other parts of the Features API along with the filter (parsed by `cql2-rs`) requires a fair amount of interweaving with `cql2-rs` (see [implementation of `cql_where()` in tipg](https://github.com/developmentseed/tipg/blob/main/tipg/collections.py#L428)). In fact, the treatment of `bbox`, `properties`, `datetime`, etc _are_ just additional cql2 clauses. So while it's a nice separation of concern to say "`cql2-rs` is responsible for parsing of the `filter` parameter only" in practice, the rest of the query parameters are already tightly linked to cql2, the `filter` parameter, and `cql2-rs`.

**Which project?**
There's 3 reasonable places to put something like this:
- **In `cql2-rs`**. As mentioned, an implementation of this will end up being tightly coupled to `cql2-rs`. It could also be argued that this is just a "helper" function that wraps existing `.to_sql()` already here. It could be added with no additional dependencies.
- **In `tipg`**. An obvious choice, but my initial use case is actually to use this _without_ tipg and independently of an actual full-fledged Features API (I would be using this to generate new `VIEW` statements). Adding it to `tipg` might also bind it closer to a single database or Features API implementation.
- **A new project**. Being just a "helper" to `cql2-rs`, and frankly not being that big, it seems a little unnecessary to be a completely independent project with all the overhead that comes with that. It would also require `tipg` to have an additional dependency and the required version tracking.

Personally, I would lean towards putting it in here, with a HUGE caveat: I'm not a Rust programmer, so I would have to implement it in Python.

**Which parts of OGC Features could be reasonable implemented?**
I think as a first pass I would basically rip out the current implementation from `tipg`. So notably no Part 2 (CRS by Reference).

I do **not** think this should construct the whole "tile" query `ST_AsMVT`. Tile serving in a performant way is hard, and this is better left up to a specific API.

**Could this be made reasonably backend independant?**
Yes (I say with 90% confidence). I believe all the needed SQL statements including `ST_` functions needed are the same between say Postgres and DuckDB. Definitely more thought needed, and if there's a glaring reason why this could not be made DB independant, please say!

**What would be the minimum goals/acceptance criteria?**
I think the initial goal would be to replace a big chunk of code in `tipg`, focusing on the query parameters that are actually part of OGC Features. Followed by adding other parameters and then ensuring DB independence.

I think it's worth putting in writing that this would do no DB-side validation. This script (as `cql2-rs` currently does) would take `properties` and other parameters and assume that they are valid column names. This would do no DB connections.

---

Those are just my thoughts for now. Open to any feedback or discussion around this. Thanks!

cc @vincentsarago @bitner as you might be closest to these 2 projects.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing .to_sql() implementation in cql2-rs and tipg's cql_where() in tipg/collections.py. Compare how OGC Features query parameters such as filter, bbox, properties, and datetime are handled. Done means agreed acceptance criteria and a backend-independent standalone SQL WHERE generator, without DB connections or tile-query construction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.