developmentseed / developmentseed/cng-formats-benchmark
Resolution-scenario latency for flat/stock-router stores is cache-warmed by fixed test ordering, not a cold measurement
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 3
Description
## The problem: resolution-scenario latency for the stock /zarr router is cache-warmed by test ordering, not a cold-request measurement
Found while sanity-checking whether `display_res_720m_latency` (972 ms, `sentinel1-zarr`, a flat no-multiscale store) was even physically plausible for the amount of data a coarse tile over a flat store should need to touch.
Verified directly: the store's native VH array is one 382,288,981-byte shard (`chunk_shape=[1098,1098]`, `shard_shape: auto` rounds up to the full array extent). A correct, complete, cold read of that array (plain zarr + fsspec, fully instrumented to confirm every byte came from S3, no shortcuts) takes **7.02s**. Calling `titiler.xarray.io.Reader.tile()` directly, cold, isolated, no FastAPI app involved, for the exact same z=7/x=65/y=46 tile the benchmark measured, takes **6.72s** -- consistent with the array-level baseline, and 7x the reported 972ms.
The gap is explained by `display_tiles.py`'s resolution-scenario set (`select_zarr_resolution_tiles`, `#116`/`#117`/`#121`): `target_resolutions` (`[20, 60, 120, 360, 720]`) are always requested in the same fixed, ascending order, against the *same* variable in the store, within *one* reader/process per run. By the time the 720m request fires, the four finer-zoom requests immediately before it (20m, 60m, 120m, 360m) have already pulled overlapping native chunks for the same ground area into cache (zarr's own chunk cache and/or the underlying obstore/HTTP connection reuse). The measured 972ms reflects a warm cache built by the scenario set's own prior requests, not the cost a real client jumping straight to a 720m-equivalent zoom would actually pay.
This does **not** affect `GeoZarrReader` (`/geozarr`) the same way: it reads a small, dedicated precomputed array for the resolved level regardless of what was requested before it, so ordering doesn't change what it has to fetch. It's specific to routers/stores with no server-side multiscale awareness, i.e. exactly the `sentinel{1,2}-zarr` arms `#121` was designed to give an honest reading of -- and this ordering bias undermines that honesty for every resolution after the first one tested.
## What it should do instead
Either:
1. Randomise (or otherwise vary) the order `display_target_resolutions` are requested in per run, so no resolution systematically benefits from the others' cache-warming, or
2. Clear the reader/store cache between each resolution scenario (mirroring the existing `/geozarr-cache/clear` endpoint's intent, extended to the stock router's own caching if it has an equivalent hook), or
3. At minimum, document clearly that only the *first*-requested resolution in a run's scenario set is a genuine cold-cache number for a no-multiscale store, and flag the rest as cache-influenced in the result -- so a report doesn't cite `res_720m` as if it were independent of `res_20m`/`res_60m`/etc. having just run moments before it against the same array.
## Acceptance
- A test with two runs of the same flat-store arm, one with the resolutions in ascending order and one in descending (or randomised) order, shows materially different `res_720m`/`res_20m` numbers depending on which one went first -- confirming the effect is real and measurable, not example-specific.
- Whichever fix lands, document which resolutions in a result are cold vs. cache-influenced.
## Related
#121 (introduced the resolution-scenario scenario set this ordering effect lives in). #126 (a different discrepancy in the same investigation, `GeoZarrReader`'s app-layer overhead -- unaffected by this, confirmed unaffected since `GeoZarrReader` doesn't do full-array scans regardless of request order). CNES study: 2026-08-13, `sentinel1-zarr`/`T31TEH` full-scene (0% nodata) run, direct-`Reader.tile()` vs measured-benchmark comparison.
Contributor guide
Research direction
Start in display_tiles.py, especially select_zarr_resolution_tiles and the target-resolution scenario set using display_target_resolutions. Run the same flat-store arm with ascending and descending resolution orders, compare res_720m and res_20m timings, and document which results are cold versus cache-influenced. Done means the acceptance comparison demonstrates the ordering effect and the chosen fix or limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100