apache / apache/sedona-spatialbench
Run the geography suite in the benchmark and commit geography ground-truth answers
- Dominant language
- Rust
- Stars
- 48
- Forks
- 23
- Avg merge
- 1d 50m
- Merged PRs (30d)
- 4
Description
Follow-up to #131, which adds the geography query suite as SQL and docs only. The benchmark harness, the CI matrix and `benchmark/answers/` are untouched by it, so the geography queries are defined but never timed or verified.
## Why the two suites need separate answers
`verify_results.py` resolves ground truth per *query*: `benchmark/answers/sf/q.parquet`. The geography suite reuses the query IDs `q1..q12` but returns **different values for the same IDs** — a 0.45° planar radius and a 50 000 m geodesic radius simply select different rows. So the answers cannot be shared, and a per-suite (or per-engine) answer path is needed rather than a flat per-query one.
Concretely, at SF1: Q1 returns a different 100 rows, Q9's areas are m² instead of square degrees, and Q7's `line_distance_m` is a real geodesic length rather than degrees divided by `0.000009`. Q2 (541) and Q4 (258 zones, identical counts and order) happen to agree with the planar answers, which makes them useful cross-checks but does not generalise.
## Work involved
1. **Runner** — teach `benchmark/run_benchmark.py` to select the suite. Either a new engine key (`sedonadb_geography`, which keeps the existing `__result.csv` dump naming working unchanged) or a `--suite {geometry,geography}` flag. `get_sql_queries()` currently hardcodes the two geometry dialect classes.
2. **Answers** — `benchmark/answers/geography/sf{1,10}/q.{csv,parquet}`, generated the same way as the existing ones, plus a note in `benchmark/answers/README.md` explaining why the two sets differ.
3. **Verify** — `verify_results.py` needs a per-engine (or per-suite) answer-directory mapping, and its `LIMIT_QUERIES` boundary-tie set should be re-derived for the geography suite rather than assumed identical.
4. **CI** — a geography job block in `.github/workflows/benchmark.yml`, mirroring the existing per-query matrix. Note the workflow already triggers on `spatialbench-queries/**`, so #131 runs the full geometry matrix without exercising any of the new SQL.
5. **Summarize** — `summarize_results.py` has a hardcoded engine list and icon map.
`QUERY_COUNT = 12` is duplicated across `run_benchmark.py`, `verify_results.py`, `generate_answers.py` and `summarize_results.py`, and the CI matrix lists `q1..q12` literally per engine; the geography suite carries 11 queries (no `ST_KNN` yet — apache/sedona-db#1086, apache/sedona#3178), so whichever mechanism is chosen has to tolerate a suite with a gap rather than assume a contiguous 1..N.
## Blocked on
Two SedonaDB defects would corrupt any answers generated today:
- **apache/sedona-db#1084** — Q9 cannot complete: a geography scalar function over geography spatial-join output fails with `Array input must not be empty`.
- **apache/sedona-db#1085** — one zone polygon matches points anywhere on Earth, inflating Q4, Q6, Q10 and Q11.
See also #133 for committing the notebook's executed outputs, which is blocked on the same two.
Contributor guide
Research direction
Read benchmark/run_benchmark.py, verify_results.py, generate_answers.py, summarize_results.py, and .github/workflows/benchmark.yml to trace suite selection, answer lookup, summaries, and CI. Confirm apache/sedona-db#1084 and #1085 are resolved before generating geography answers. Done means the 11-query geography suite runs, verifies against separate answers, summarizes correctly, and has a CI matrix without assuming contiguous query IDs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python, sql
- Domain
- ci-cd, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100