developmentseed / developmentseed/cng-formats-benchmark
Aggregation (consolidation) arm: measure the tier lever for the sub-Tier-2 products
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 3
Description
## Motivation
For three of the benchmarked products (SWOT Raster100m, SWOT LakeSP, S2 Let-it-snow) the measured verdict is the same: the cloud-native output is **sub-Tier-2 per delivered unit**, so the object-grouping lever is not tiling, sharding or row-group size, it is **aggregation across units**. The CNES report states that recommendation (D2 ch. 6.4, 6.5, 7.3) but supports it with a **first-order estimate**, not a measurement: for example "order 60 to 80 MiB per UTM zone per cycle" is the per-tile COG size times a tile count, not a consolidated object that was written and read.
The harness converts one source component into one output object, so the aggregation lever is currently the one recommendation it cannot verify. This is the missing arm that turns the tier advice from arithmetic into evidence.
## Ask
An **aggregation (consolidation) arm**: convert *N* source products into **one** output object per format, then measure it with the existing metric set.
- **Raster (COG / GeoZarr).** Mosaic the granules of a group into one object: for SWOT Raster100m the natural group is **one UTM zone, one cycle**; for LIS a tile across dates (which for GeoZarr means a `time` dimension, the time-cube shape already identified for S2).
- **Vector (GeoParquet / FlatGeobuf).** Concatenate the passes of a group into one file, with the row groups sized so a bbox query still prunes.
- **Point cloud (COPC).** Merge adjacent tiles into one octree, which is the CO3D question as much as the SWOT one.
- **Config surface.** A grouping key on the arm (something like `aggregate.by: utm_zone+cycle` or a regex capture over the product name) plus the group size, so a run can sweep "how many units per object" and show where the group crosses the Tier 2 and Tier 3 floors.
- **What it must report.** The aggregated object's size and tier verdict, the **write cost of consolidating**, and above all a **subset read on the aggregate**: the whole point is that a bigger object stays range-addressable, so a random bbox read on the consolidated object must be compared against the same read on the per-unit objects. If aggregation clears the tier but ruins partial access, that is the finding.
## Acceptance
- An aggregation mode selectable per arm, unit-tested on synthetic multi-unit fixtures for at least the raster and vector paths.
- Committed runs for SWOT Raster100m (per UTM zone per cycle) and SWOT LakeSP (per cycle), reporting object count and size before and after, tier verdicts, write cost and subset-read latency both ways.
- The D2 estimate replaced by a measured number.
## Touch points
`src/cng_benchmark/runner.py` (product grouping), the format adapters (`cog.py`, `geozarr.py`, `geoparquet.py`, `copc.py`), `configs/benchmarks/`, `docs/configuration.md`.
## Related
- CNES study: D2 ch. 6.4, 6.5, 7.3; `consolidated-metrics.md` section 4 ("the other size regime: small products need aggregation").
- Feeds the LIS and CO3D issues in this batch, where the delivered unit is also sub-tier.
Contributor guide
Research direction
Start with product grouping in src/cng_benchmark/runner.py, then inspect cog.py, geozarr.py, geoparquet.py, and copc.py for the format-specific paths. Review configs/benchmarks/ and docs/configuration.md to understand the configuration surface and existing metric reporting. Done means synthetic raster and vector fixtures cover aggregation, and committed SWOT runs report before/after objects, tier verdicts, consolidation cost, and subset-read latency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100