obs(api): no metrics on global expiration ZSET size or evictor sweep latency
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Background
`sandbox:storage:global:expiration` is a single global ZSET shared by every
sandbox across all teams. The evictor calls `ExpiredItems` every 50 ms
(`pollInterval`), which issues a `ZRANGEBYSCORE` against this key on every tick.
Problem
There are currently no metrics to observe:
- ZSET cardinality over time — no way to know whether
`sandbox:storage:global:expiration` is growing unboundedly, shrinking after
orphan sweeps (#3567), or stable at steady state. - Evictor sweep duration — no way to know how long each
`ZRANGEBYSCORE + MGET pipeline` takes, or whether sweep latency is
increasing as the ZSET grows.
Without these two signals it is impossible to:
- Set an alert before the ZSET grows large enough to cause latency spikes
- Verify that orphan cleanup (#3567) or future sharding fixes are actually
reducing cardinality - Correlate Redis command latency spikes with evictor tick cost
Production evidence
Observed 2026-08-14 09:00 – 09:30 (Redis big-key scan):
| Key | Type | Memory | Elements |
|---|---|---|---|
| `sandbox:storage:global:expiration` | zset | 3,137,528 B (~3 MB) | 22,458 |
| `sandbox:storage:{65380065...}:index` | set | 588,824 B | 9,535 |
| `template:info:{fnqz33l5lhnpqiephgbl}:default` | string | 12,304 B | 1 |
At `pollInterval = 50 ms` and N API allocations, the evictor issues
20 × N `ZRANGEBYSCORE` calls per second against the 22 k-member ZSET.
This load is currently invisible — there is no metric showing either the
key size trend or the per-sweep cost.
Observability gap
| Signal | Metric name (proposed) | Collection point |
|---|---|---|
| ZSET cardinality | `api.redis_storage.expiration_index.size` (gauge) | Once per heal pass (`healExpirationIndex`, every 5 min) |
| Evictor sweep duration | `api.redis_storage.expiration_index.sweep_duration` (histogram, ms) | Per `ExpiredItems` call |
Why a gauge for size, not sampled on every evictor tick?
`global:expiration` is already the key being diagnosed as a high-frequency
hot key. Sampling `ZCARD` on every 50 ms evictor tick would add 20 × N
extra Redis calls per second — worsening the problem while observing it.
A single `ZCARD` piggybacked on the existing heal pass (every 5 min,
same cadence as #3604) adds negligible overhead and is sufficient for
capacity planning and alerting.
Why sweep duration matters
`ZRANGEBYSCORE` is O(log N + K) where N = 22,458 and K = batch size (256).
As N grows the per-sweep cost grows. Correlating sweep duration with ZSET
size over time makes the relationship visible before it becomes a latency
incident.
Related
- #3566 / #3567 — stale team-index entries that may be inflating
`global:expiration` via orphaned ZSET members - #3602 / #3603 — `cjson.decode` blocking Redis event loop on
`startTransitionScript` (separate hot path) - #3604 — per-team index SET size histogram (same observability series)
- #3591 — RFC: evaluate DragonflyDB as caching backend; ZSET size and
sweep latency metrics would provide the baseline needed to evaluate
whether a backend switch is warranted
/cc @jakubno @dobrac @ValentaTomas
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the healExpirationIndex pass and the ExpiredItems call in the evictor, then identify the existing metrics conventions used by the API storage code. Add the proposed expiration_index.size gauge during the heal pass and sweep_duration histogram per ExpiredItems call, and verify both signals are emitted with the documented names and units.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, redis
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100