cockroachdb / cockroachdb/cockroach
sql/catalog/lease: dropped descriptors are retained in the lease manager (memory leak)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The descriptor lease manager retains the in-memory state of dropped descriptors for the lifetime of the process. When a descriptor (e.g. a table) is dropped, its decoded, immutable descriptor graph stays reachable inside the lease manager even after all active leases are gone. On long-running nodes under a workload that creates and drops many uniquely-named descriptors, this manifests as steady, unbounded heap growth attributed to `sql/catalog/lease.storage.acquireBatch` (the descriptor-construction path), which is never released.
Notably, the `leased-descriptors` memory monitor does *not* reflect this: its bound account is correctly shrunk when a lease is released, so the retained memory is unaccounted — the Go objects simply remain reachable.
There are three independent retention paths, all of which trigger deterministically on drop:
1. **The name cache is not evicted on drop.** A descriptor version is inserted into the lease manager's `nameCache` on acquisition, but no drop/release/purge path removes it. The only removal is lazy, inside `nameCache.get()`, when the same name is resolved again and found with a nil lease. For a uniquely-named descriptor that is never resolved again, that path is never reached, so the entry (and the descriptor it points to) lingers forever.
2. **`descriptorSet.remove` leaves a stale backing-array slot.** Removal uses `append(l.data[:i], l.data[i+1:]...)`, which shifts elements down but does not nil the now-unused tail slot of the backing array. The removed `*descriptorVersionState` remains reachable through the retained backing array even after the set is logically drained to length 0.
3. **The `descriptorState` is never deleted from `Manager.mu.descriptors`.** On drop, `purgeOldVersions(dropped=true)` marks the state offline and drains its active versions but never deletes the map entry. `refreshSomeLeases` skips offline states, and the only `delete(m.mu.descriptors, id)` sits on the lease-refresh error path, which offline states never reach. So the (now-empty) state — and, via (2), the backing array it owns — persists indefinitely.
**To Reproduce**
1. Start a CockroachDB node.
2. Repeatedly, in a loop: `CREATE TABLE` with a unique name, read from it (so a lease is acquired), then `DROP TABLE`.
3. Observe an in-use heap profile over time: memory under `sql/catalog/lease.storage.acquireBatch` / `tabledesc.BuildImmutableTable` grows monotonically and is never released, while the `leased-descriptors` monitor stays low. The count of entries in the lease manager's descriptor map also grows without bound.
**Expected behavior**
When a descriptor is dropped and no longer has any active leases, all of its in-memory state in the lease manager (name-cache entry, version state, and descriptor-map entry) should be released so it can be garbage collected.
**Additional data / screenshots**
Proposed fixes, one per retention path:
1. **Name cache:** evict the name-cache entry whenever a version is removed from the active set (in both `removeInactiveVersions` and the `release()` removal path). `nameCache.remove` is already ID-guarded, so it's a no-op if a newer version replaced the entry.
2. **Backing-array slot:** in `descriptorSet.remove`, shift the tail down and nil the vacated slot before reslicing, so the backing array no longer pins the removed version.
3. **Descriptor map:** add a bounded periodic reaper (driven by the existing lease-refresher loop) that removes offline descriptor states with no remaining active versions from `Manager.mu.descriptors`. It also sweeps any versions that were dereferenced late (e.g. a version dropped while still leased whose stored lease was cleared before its refcount reached zero).
The expensive part (the decoded descriptor graph) is freed eagerly on drop by fixes (1) and (2); fix (3) removes the small leftover state on a timer.
**Environment:**
- CockroachDB version: affects all currently supported versions (the mechanism has been present for several releases; not a recent regression)
- Server OS: any
- Client app: n/a (server-side)
**Additional context**
Impact: unbounded server memory growth on nodes with long uptime and a high rate of dropped, uniquely-named descriptors, up to and including OOM. A node restart temporarily clears the accumulated memory.
Related: #150279 (eviction policy for *live* leased descriptors — a separate enhancement; it would not address this leak because the bound account is already shrunk on release), #79355 (descriptor byte-size under-accounting), #80107 (name cache not timestamp-aware).
Jira issue: CRDB-68167
Epic CRDB-65516
Contributor guide
Assessment
This issue has not been assessed yet.