developmentseed / developmentseed/cng-sandbox
Spike: evaluate yosegi (Pyramid GeoParquet) for large-vector streaming
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 2h 21m
- Merged PRs (30d)
- 3
Description
## Background
[yosegi](https://github.com/Kanahiro/yosegi) is a Python tool that converts a GeoParquet (or any GDAL/OGR vector) into a **Pyramid GeoParquet**: a single file with two appended columns (`bbox` STRUCT + `zoomlevel` INT32) and STR-packed row groups, so a browser can range-read tiles directly from object storage via DuckDB-WASM. No tile server, no Postgres.
For a sandbox whose current vector path is `GeoParquet → Postgres → tipg → MVT`, yosegi offers a parallel path that scales much better for very large vectors (Overture-scale, 100s of MB to GB).
## Why it might fit us
- We already convert vectors to GeoParquet and already host raster artifacts (COGs) on R2 — Pyramid GeoParquet drops into the same pattern.
- Frontend already uses deck.gl for raster; yosegi's reference renderer is `GeoArrowScatterplotLayer` / DuckDB-WASM. Same stack.
- Removes server tile rendering for the largest, hardest-to-serve datasets.
- yosegi claims STR-pack touches 27–37% fewer row groups than Hilbert sort for tile-bbox queries on heavy-tailed feature data, with an analytic upper bound on per-RG extent.
## Proposed shape (not a final design)
Keep tipg as the default for small / queryable vectors. Add a second ingestion path for large vectors:
1. **Ingestion**: after producing GeoParquet, if size/feature-count exceeds a threshold (e.g. >100 MB or >1M features), run `yosegi` to emit `*.pyramid.parquet` and upload to R2 alongside COGs.
2. **Registration**: record a new dataset variant (`vector_pyramid` or similar) on the dataset row with the R2 URL.
3. **Frontend**: new layer type that loads DuckDB-WASM, range-reads the pyramid by current tile bbox + zoom, hands GeoArrow batches to deck.gl.
## Tradeoffs to validate in the spike
- **Loses server-side attribute filtering** that tipg provides. For datasets where users want SQL-style queries, tipg is still the right path.
- **Bundle weight**: DuckDB-WASM is a non-trivial frontend dependency. Should be lazy-loaded.
- **Ingestion time + storage**: pyramid build is extra work; only worth it past a size threshold. Need to measure.
- **GDAL/OGR dep**: yosegi is a Python package; need to confirm it slots cleanly into the existing `ingestion/` uv environment without conflicting native deps.
- **Browser perf**: need to confirm DuckDB-WASM + GeoArrowScatterplotLayer on a real Overture-scale dataset behaves well on a typical user's machine.
## Spike scope
- [ ] Install `yosegi` in `ingestion/` venv, confirm it runs against an existing example GeoParquet
- [ ] Generate a pyramid for one large example dataset (Overture or similar) and upload to R2
- [ ] Prototype a minimal frontend page that loads it via DuckDB-WASM + deck.gl
- [ ] Measure: ingestion time delta, output size ratio, time-to-first-render in browser, RGs touched per tile
- [ ] Decide: green-light a real integration, shelve, or revisit later
## Out of scope for spike
- Replacing tipg
- UI for choosing tipg vs pyramid path
- Auth / signed URLs for pyramid files
Contributor guide
Assessment
This issue has not been assessed yet.