e2b-dev / e2b-dev/runtime

[Feature] Bound template mmap cache size and add memory-pressure eviction to prevent unbounded RSS growth

Open
#2,957 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Problem

In self-hosted deployments with sustained sandbox workloads, the Orchestrator's RSS grows monotonically and is never reclaimed automatically.
Observed metrics: RSS increased from ~200 MB to ~37 GB within 48 hours.
Most memory consumption comes from Pss_File, which is occupied by mmap-backed template caches.

Root Cause Analysis
  1. Template cache has no capacity limit
    The template cache in pkg/sandbox/template/cache.go is initialized only with a fixed TTL, without maximum capacity restriction:
    cache := ttlcache.New(
    ttlcache.WithTTLstring, Template, // hardcoded 25 h
    // no WithCapacity — cache grows without bound
    )
    getTemplateWithFetch calls cache.GetOrSet, which resets TTL on every access.
    Templates used by long-running sandboxes will never be evicted.
    Each cache entry holds an mmap.MMap handle, locking file pages in RAM until eviction.
  2. Build cache eviction only checks disk usage (no memory pressure handling)
    The eviction loop startDiskSpaceEviction in pkg/sandbox/build/cache.go only judges by disk usage (BuildCacheMaxUsagePercentage, default 85%).
    If disk usage is under threshold but host memory is exhausted, no cache eviction will be triggered.
    The build diff cache (DiffStore) and template mmap cache are two independent subsystems; neither supports memory-based eviction.
Proposed Changes

Add capacity limit for template cache
Add ttlcache.WithCapacity to limit template cache size. Use an environment variable (e.g. TEMPLATE_CACHE_MAX_ENTRIES) for configuration, following existing BuilderConfig style. Evict entries by LRU policy when reaching the cap.
Add memory pressure detection for build cache eviction
Extend startDiskSpaceEviction:
Read system memory status via unix.Sysinfo
Trigger deleteOldestFromCache when free memory is lower than a configurable threshold (e.g. CACHE_MIN_FREE_MEMORY_MB)
Make template TTL configurable
Replace the hardcoded 25h templateExpiration with an environment variable. The 25h default works for cloud environments, but is too long for memory-limited self-hosted nodes.

Notes

All above changes work as safety valves only. They will not affect normal running behavior on nodes with sufficient resources.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/sandbox/template/cache.go and the getTemplateWithFetch path, then inspect pkg/sandbox/build/cache.go and startDiskSpaceEviction. Trace the existing BuilderConfig-style environment configuration and cache eviction behavior. Done means template capacity and TTL are configurable, and build-cache eviction can respond to low free memory as well as disk usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.