developmentseed / developmentseed/cng-formats-benchmark

titiler.xarray's open_zarr cache is unbounded (functools.cache, no eviction) -- OOMKilled a standalone pod after 3 requests

Open
#129 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4
Forks
0
Avg merge
1d 7h
Merged PRs (30d)
3

Description

## The problem: `titiler.xarray.io.open_zarr` caches every opened dataset forever, with no eviction -- a real memory leak

Found while live-testing `/zarr` vs `/geozarr` HTTP response times for a single tile against a standalone titiler pod (no benchmark runner involved, just the tiler service).

```python
# titiler/xarray/io.py
from functools import cache

@cache
def open_zarr(
src_path: str,
group: str | None = None,
decode_times: bool = True,
decode_coords: str = "all",
infer_region: bool = True,
**kwargs: Any,
) -> xarray.Dataset:
...
```

`functools.cache` is `lru_cache(maxsize=None)`: unbounded, no eviction, ever. Every distinct `(src_path, group, ...)` combination this function is ever called with keeps its opened `xarray.Dataset` (and whatever it holds onto -- coordinate arrays, dask graph state, file handles) resident in process memory for the lifetime of the pod. A long-running titiler deployment serving the stock `/zarr` router across many different stores over its lifetime -- exactly this study's own usage pattern, and any real deployment's -- accumulates memory without bound.

Confirmed empirically, not just from reading the code: launched a standalone titiler pod (workers=1, no benchmark runner) and sent 3 sequential single-tile requests (a `/zarr` z6 request, a `/geozarr` z6 request, then a `/geozarr` z7 request against a store already open). The pod was OOMKilled (exit 137) partway through:

```
Warning Evicted kubelet The node was low on resource: memory.
Threshold quantity: 100Mi, available: 8512Ki.
Container titiler was using 549300Ki, request is 0,
has larger consumption of memory.
```

549 MB resident after 3 requests against 2 stores, no explicit memory limit set on the container (`request is 0`), so nothing stopped it from growing until the *node* ran out of memory and evicted it. The underlying store's native VH array is 382 MB -- consistent with `open_zarr`'s cached `xarray.Dataset` holding something close to that per store, permanently, per distinct store ever requested.

This is very likely the root cause (or a major contributor) behind the `pool-general` `MemoryPressure` incidents that recurred repeatedly across this campaign whenever multiple GeoZarr display-metric runs shared that node -- not (only) N-pods-competing-for-a-fixed-budget contention, but each individual `/zarr`-serving pod's own memory growing without bound as it's asked to serve more distinct stores.

`titiler.eopf`'s equivalent (`GeoZarrReader`'s `open_dataset`, used by `/geozarr`) does **not** have this problem: it's `@lru_cache(maxsize=DATASET_CACHE_MAXSIZE)`, a bounded cache with real eviction.

## What it should do instead

Bound `open_zarr`'s cache the same way `titiler.eopf` already does: `@lru_cache(maxsize=N)` for some sane `N` (or a size-aware cache keyed on estimated dataset memory footprint, if store sizes vary a lot), so old, no-longer-needed datasets get evicted rather than accumulating forever. `cng_benchmark_tiler` (the benchmark's own tiler app, `main.py`) wraps `titiler.xarray.factory.TilerFactory` as-is and doesn't override this -- the fix belongs in whichever of `titiler.xarray` (upstream) or this repo's own tiler app is the right place to patch it, worth checking which.

## Acceptance

- A titiler pod serving `/zarr` against N distinct stores, N large enough to exceed a fixed memory budget if unbounded, does not OOM -- memory plateaus once the cache reaches its bound.
- A repeat request against a store evicted from cache still works (falls through to a fresh open), just pays the re-open cost, not a crash.

## Related

#126 (the `/geozarr` app-layer latency gap -- distinct issue, `GeoZarrReader`'s own cache is already bounded, this is specific to the stock `/zarr` router's `open_zarr`). #128 (resolution-scenario cache-warming from test ordering -- a *different* cache, the same underlying `open_zarr`/dataset cache this issue is about, but that issue is about ordering bias in what's already cached, this one is about the cache never releasing anything at all). CNES study: 2026-08-13, standalone titiler pod, 3-request OOM reproduction.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting titiler/xarray/io.py and cng_benchmark_tiler's main.py to determine whether the cache belongs upstream or in this app, then compare titiler.eopf's bounded GeoZarrReader cache. Reproduce the sequential multi-store requests if possible and verify that the selected cache bound plateaus memory, evicts old datasets, and still allows an evicted store to reopen successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.