e2b-dev / e2b-dev/runtime

obs(api): no metrics on global expiration ZSET size or evictor sweep latency

Open
#3,605 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Background

`sandbox:storage:global:expiration` is a single global ZSET shared by every
sandbox across all teams. The evictor calls `ExpiredItems` every 50 ms
(`pollInterval`), which issues a `ZRANGEBYSCORE` against this key on every tick.

Problem

There are currently no metrics to observe:

  1. ZSET cardinality over time — no way to know whether
    `sandbox:storage:global:expiration` is growing unboundedly, shrinking after
    orphan sweeps (#3567), or stable at steady state.
  2. Evictor sweep duration — no way to know how long each
    `ZRANGEBYSCORE + MGET pipeline` takes, or whether sweep latency is
    increasing as the ZSET grows.

Without these two signals it is impossible to:

  • Set an alert before the ZSET grows large enough to cause latency spikes
  • Verify that orphan cleanup (#3567) or future sharding fixes are actually
    reducing cardinality
  • Correlate Redis command latency spikes with evictor tick cost

Production evidence

Observed 2026-08-14 09:00 – 09:30 (Redis big-key scan):

Key Type Memory Elements
`sandbox:storage:global:expiration` zset 3,137,528 B (~3 MB) 22,458
`sandbox:storage:{65380065...}:index` set 588,824 B 9,535
`template:info:{fnqz33l5lhnpqiephgbl}:default` string 12,304 B 1

At `pollInterval = 50 ms` and N API allocations, the evictor issues
20 × N `ZRANGEBYSCORE` calls per second against the 22 k-member ZSET.
This load is currently invisible — there is no metric showing either the
key size trend or the per-sweep cost.

Observability gap

Signal Metric name (proposed) Collection point
ZSET cardinality `api.redis_storage.expiration_index.size` (gauge) Once per heal pass (`healExpirationIndex`, every 5 min)
Evictor sweep duration `api.redis_storage.expiration_index.sweep_duration` (histogram, ms) Per `ExpiredItems` call
Why a gauge for size, not sampled on every evictor tick?

`global:expiration` is already the key being diagnosed as a high-frequency
hot key. Sampling `ZCARD` on every 50 ms evictor tick would add 20 × N
extra Redis calls per second — worsening the problem while observing it.
A single `ZCARD` piggybacked on the existing heal pass (every 5 min,
same cadence as #3604) adds negligible overhead and is sufficient for
capacity planning and alerting.

Why sweep duration matters

`ZRANGEBYSCORE` is O(log N + K) where N = 22,458 and K = batch size (256).
As N grows the per-sweep cost grows. Correlating sweep duration with ZSET
size over time makes the relationship visible before it becomes a latency
incident.

Related

  • #3566 / #3567 — stale team-index entries that may be inflating
    `global:expiration` via orphaned ZSET members
  • #3602 / #3603 — `cjson.decode` blocking Redis event loop on
    `startTransitionScript` (separate hot path)
  • #3604 — per-team index SET size histogram (same observability series)
  • #3591 — RFC: evaluate DragonflyDB as caching backend; ZSET size and
    sweep latency metrics would provide the baseline needed to evaluate
    whether a backend switch is warranted

/cc @jakubno @dobrac @ValentaTomas

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the healExpirationIndex pass and the ExpiredItems call in the evictor, then identify the existing metrics conventions used by the API storage code. Add the proposed expiration_index.size gauge during the heal pass and sweep_duration histogram per ExpiredItems call, and verify both signals are emitted with the documented names and units.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, redis
Domain
backend, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.