developmentseed / developmentseed/titiler

RSS grows well past GDAL_CACHEMAX under concurrent tile reads; MALLOC_ARENA_MAX reduces it by ~27%

Open Beginner friendly
#1,477 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
254
Avg merge
1d 16h
Merged PRs (30d)
9

Description

Sharing a measurement in case it is useful for the performance tuning docs, which cover GDAL caching but not allocator behavior.

## What we saw

A single-worker TiTiler container serving raster tiles grew from roughly 0.3 GB to 1.78 GB of RSS over 20 hours and plateaued near its 2 GB limit. `GDAL_CACHEMAX` was pinned at 200 MB throughout. `smaps_rollup` showed `Private_Dirty` equal to `Anonymous` at 1.63 GB, so the retention was allocator heap rather than the GDAL block cache.

The mechanism appears to be that rasterio reads are synchronous and run on the anyio threadpool, and glibc gives each thread its own malloc arena (8 times the core count by default), which retains freed tile buffers.

## Measurement

2880 identical raster tile requests at concurrency 16, all HTTP 200, on a 4 core host. The container was force recreated between arms.

| `MALLOC_ARENA_MAX` | before | after | growth |
| --- | --- | --- | --- |
| 32 (glibc default for 4 cores) | 157 MB | 329 MB | +172 MB |
| 2 | 156 MB | 281 MB | +125 MB |

About 27% less growth for one environment variable. One run per arm, both cold started from a fresh container with matching baselines, so treat the exact percentage as indicative.

## What it is and is not

The arena cap slows the growth, it does not remove it. Roughly 125 MB survives it, so the rest is elsewhere: pymalloc retention, rio-tiler caches, or intra-arena fragmentation. We size a memory limit and treat an OOM restart as the real backstop, with the arena cap as a mitigation rather than a fix.

Fewer arenas can increase malloc contention on many core hosts serving heavy concurrent reads, so it is a tradeoff rather than a free win.

Would a short note in the performance tuning docs be useful? I am happy to open a PR. Equally happy to hear this is environment specific and not worth documenting.

Contributor guide

Open the contributing guide

Research direction

Start with the performance tuning documentation and its existing GDAL caching guidance. Document allocator arena retention as a separate memory consideration, including the reported MALLOC_ARENA_MAX measurements, mitigation limits, and contention tradeoff. Done means the note clearly distinguishes allocator memory from GDAL_CACHEMAX and treats the exact percentage as environment-specific.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, performance
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.