developmentseed / developmentseed/cng-formats-benchmark

Resolution-scenario latency for flat/stock-router stores is cache-warmed by fixed test ordering, not a cold measurement

Open
#128 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4
Forks
0
Avg merge
1d 7h
Merged PRs (30d)
3

Description

## The problem: resolution-scenario latency for the stock /zarr router is cache-warmed by test ordering, not a cold-request measurement

Found while sanity-checking whether `display_res_720m_latency` (972 ms, `sentinel1-zarr`, a flat no-multiscale store) was even physically plausible for the amount of data a coarse tile over a flat store should need to touch.

Verified directly: the store's native VH array is one 382,288,981-byte shard (`chunk_shape=[1098,1098]`, `shard_shape: auto` rounds up to the full array extent). A correct, complete, cold read of that array (plain zarr + fsspec, fully instrumented to confirm every byte came from S3, no shortcuts) takes **7.02s**. Calling `titiler.xarray.io.Reader.tile()` directly, cold, isolated, no FastAPI app involved, for the exact same z=7/x=65/y=46 tile the benchmark measured, takes **6.72s** -- consistent with the array-level baseline, and 7x the reported 972ms.

The gap is explained by `display_tiles.py`'s resolution-scenario set (`select_zarr_resolution_tiles`, `#116`/`#117`/`#121`): `target_resolutions` (`[20, 60, 120, 360, 720]`) are always requested in the same fixed, ascending order, against the *same* variable in the store, within *one* reader/process per run. By the time the 720m request fires, the four finer-zoom requests immediately before it (20m, 60m, 120m, 360m) have already pulled overlapping native chunks for the same ground area into cache (zarr's own chunk cache and/or the underlying obstore/HTTP connection reuse). The measured 972ms reflects a warm cache built by the scenario set's own prior requests, not the cost a real client jumping straight to a 720m-equivalent zoom would actually pay.

This does **not** affect `GeoZarrReader` (`/geozarr`) the same way: it reads a small, dedicated precomputed array for the resolved level regardless of what was requested before it, so ordering doesn't change what it has to fetch. It's specific to routers/stores with no server-side multiscale awareness, i.e. exactly the `sentinel{1,2}-zarr` arms `#121` was designed to give an honest reading of -- and this ordering bias undermines that honesty for every resolution after the first one tested.

## What it should do instead

Either:
1. Randomise (or otherwise vary) the order `display_target_resolutions` are requested in per run, so no resolution systematically benefits from the others' cache-warming, or
2. Clear the reader/store cache between each resolution scenario (mirroring the existing `/geozarr-cache/clear` endpoint's intent, extended to the stock router's own caching if it has an equivalent hook), or
3. At minimum, document clearly that only the *first*-requested resolution in a run's scenario set is a genuine cold-cache number for a no-multiscale store, and flag the rest as cache-influenced in the result -- so a report doesn't cite `res_720m` as if it were independent of `res_20m`/`res_60m`/etc. having just run moments before it against the same array.

## Acceptance

- A test with two runs of the same flat-store arm, one with the resolutions in ascending order and one in descending (or randomised) order, shows materially different `res_720m`/`res_20m` numbers depending on which one went first -- confirming the effect is real and measurable, not example-specific.
- Whichever fix lands, document which resolutions in a result are cold vs. cache-influenced.

## Related

#121 (introduced the resolution-scenario scenario set this ordering effect lives in). #126 (a different discrepancy in the same investigation, `GeoZarrReader`'s app-layer overhead -- unaffected by this, confirmed unaffected since `GeoZarrReader` doesn't do full-array scans regardless of request order). CNES study: 2026-08-13, `sentinel1-zarr`/`T31TEH` full-scene (0% nodata) run, direct-`Reader.tile()` vs measured-benchmark comparison.

Contributor guide

Open the contributing guide

Research direction

Start in display_tiles.py, especially select_zarr_resolution_tiles and the target-resolution scenario set using display_target_resolutions. Run the same flat-store arm with ascending and descending resolution orders, compare res_720m and res_20m timings, and document which results are cold versus cache-influenced. Done means the acceptance comparison demonstrates the ordering effect and the chosen fix or limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.