cockroachdb / cockroachdb/cockroach
sql: TemporaryObjectCleaner walks all databases + all schemas every 30 min to find name-prefix matches; scales poorly at many-tables clusters
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary**
`TemporaryObjectCleaner.doTemporaryObjectCleanup` walks every database and
every schema in each database on every cleanup tick, then filters in Go for
schemas whose name matches the `pg_temp_` prefix. The filter
could be expressed as a single namespace-prefix scan, but the current code
materializes the full namespace fan-out across all databases.
Surfaced during 100K-table testing for the 1M-tables project (parent epic
CRDB-62272), as part of an audit for code with the same shape as the
auto-stats refresher's `getApplicableTables` (see #171192). This one is
**recurring every 30 min** rather than one-shot at server startup, and runs
**on every node** independently.
**Code references**
- [pkg/sql/temporary_schema.go#L725-L777](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L725-L777) — `doTemporaryObjectCleanup`:
- line 743: `descsCol.GetAllDatabases(ctx, txn.KV())` — read all DBs
- line 748: `dbs.ForEachDescriptor(...)` — outer loop
- line 755: `descsCol.GetAllSchemasInDatabase(ctx, txn.KV(), db)` — read all schemas in DB
- line 760-776: walk every namespace entry, filter in Go via `temporarySchemaSessionID`
- [pkg/sql/temporary_schema.go#L150-L168](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L150-L168) — `temporarySchemaSessionID`: the Go-side filter, just a `pg_temp_` prefix check + sessionID parse
- [pkg/sql/temporary_schema.go#L47-L52](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L47-L52) — `sql.temp_object_cleaner.cleanup_interval`, default 30 min
- [pkg/sql/temporary_schema.go#L855-L876](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L855-L876) — the per-node cleanup loop
**Cost profile**
Per node, per 30-min cycle:
- 1 KV read of all database descriptors
- N KV reads of all namespace entries (one per database)
- Materialization in Go of every schema across the cluster
- A Go-side prefix check on each entry that, in the common case, yields zero matches
In a cluster with no temporary schemas, this is pure overhead. A 9-node
1M-table cluster pays this overhead 9 times every 30 min — and we have
no observability on how long the walk itself takes.
**Suggested direction**
Not prescriptive. The obvious shape:
- A dedicated `system.namespace` query / catalog API that returns only
entries whose name begins with `pg_temp_`. Pushes the filter to KV
instead of Go. Could be a `ScanNamespaceEntriesByNamePrefix` on the
catalog reader, or an indexed query on the namespace table.
- The wait-interval check (line 766) and the session-ID parse (line 769)
remain in Go and operate only on the small filtered set.
**Related**
- #171192 — `getApplicableTables` same-shape issue in auto-stats refresher
- Epic CRDB-62272
Jira issue: CRDB-64373
Epic CRDB-58778
Contributor guide
Assessment
This issue has not been assessed yet.