developmentseed / developmentseed/titiler
RSS grows well past GDAL_CACHEMAX under concurrent tile reads; MALLOC_ARENA_MAX reduces it by ~27%
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 254
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
Sharing a measurement in case it is useful for the performance tuning docs, which cover GDAL caching but not allocator behavior.
## What we saw
A single-worker TiTiler container serving raster tiles grew from roughly 0.3 GB to 1.78 GB of RSS over 20 hours and plateaued near its 2 GB limit. `GDAL_CACHEMAX` was pinned at 200 MB throughout. `smaps_rollup` showed `Private_Dirty` equal to `Anonymous` at 1.63 GB, so the retention was allocator heap rather than the GDAL block cache.
The mechanism appears to be that rasterio reads are synchronous and run on the anyio threadpool, and glibc gives each thread its own malloc arena (8 times the core count by default), which retains freed tile buffers.
## Measurement
2880 identical raster tile requests at concurrency 16, all HTTP 200, on a 4 core host. The container was force recreated between arms.
| `MALLOC_ARENA_MAX` | before | after | growth |
| --- | --- | --- | --- |
| 32 (glibc default for 4 cores) | 157 MB | 329 MB | +172 MB |
| 2 | 156 MB | 281 MB | +125 MB |
About 27% less growth for one environment variable. One run per arm, both cold started from a fresh container with matching baselines, so treat the exact percentage as indicative.
## What it is and is not
The arena cap slows the growth, it does not remove it. Roughly 125 MB survives it, so the rest is elsewhere: pymalloc retention, rio-tiler caches, or intra-arena fragmentation. We size a memory limit and treat an OOM restart as the real backstop, with the arena cap as a mitigation rather than a fix.
Fewer arenas can increase malloc contention on many core hosts serving heavy concurrent reads, so it is a tradeoff rather than a free win.
Would a short note in the performance tuning docs be useful? I am happy to open a PR. Equally happy to hear this is environment specific and not worth documenting.
Contributor guide
Research direction
Start with the performance tuning documentation and its existing GDAL caching guidance. Document allocator arena retention as a separate memory consideration, including the reported MALLOC_ARENA_MAX measurements, mitigation limits, and contention tradeoff. Done means the note clearly distinguishes allocator memory from GDAL_CACHEMAX and treats the exact percentage as environment-specific.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, performance
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100