cockroachdb / cockroachdb/cockroach

sql: TemporaryObjectCleaner walks all databases + all schemas every 30 min to find name-prefix matches; scales poorly at many-tables clusters

Open
#171,227 0 comments 0 reactions 0 assignees View on GitHub
A-many-descriptors C-enhancement O-agent T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary**

`TemporaryObjectCleaner.doTemporaryObjectCleanup` walks every database and
every schema in each database on every cleanup tick, then filters in Go for
schemas whose name matches the `pg_temp_` prefix. The filter
could be expressed as a single namespace-prefix scan, but the current code
materializes the full namespace fan-out across all databases.

Surfaced during 100K-table testing for the 1M-tables project (parent epic
CRDB-62272), as part of an audit for code with the same shape as the
auto-stats refresher's `getApplicableTables` (see #171192). This one is
**recurring every 30 min** rather than one-shot at server startup, and runs
**on every node** independently.

**Code references**

- [pkg/sql/temporary_schema.go#L725-L777](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L725-L777) — `doTemporaryObjectCleanup`:
- line 743: `descsCol.GetAllDatabases(ctx, txn.KV())` — read all DBs
- line 748: `dbs.ForEachDescriptor(...)` — outer loop
- line 755: `descsCol.GetAllSchemasInDatabase(ctx, txn.KV(), db)` — read all schemas in DB
- line 760-776: walk every namespace entry, filter in Go via `temporarySchemaSessionID`
- [pkg/sql/temporary_schema.go#L150-L168](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L150-L168) — `temporarySchemaSessionID`: the Go-side filter, just a `pg_temp_` prefix check + sessionID parse
- [pkg/sql/temporary_schema.go#L47-L52](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L47-L52) — `sql.temp_object_cleaner.cleanup_interval`, default 30 min
- [pkg/sql/temporary_schema.go#L855-L876](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/temporary_schema.go#L855-L876) — the per-node cleanup loop

**Cost profile**

Per node, per 30-min cycle:

- 1 KV read of all database descriptors
- N KV reads of all namespace entries (one per database)
- Materialization in Go of every schema across the cluster
- A Go-side prefix check on each entry that, in the common case, yields zero matches

In a cluster with no temporary schemas, this is pure overhead. A 9-node
1M-table cluster pays this overhead 9 times every 30 min — and we have
no observability on how long the walk itself takes.

**Suggested direction**

Not prescriptive. The obvious shape:

- A dedicated `system.namespace` query / catalog API that returns only
entries whose name begins with `pg_temp_`. Pushes the filter to KV
instead of Go. Could be a `ScanNamespaceEntriesByNamePrefix` on the
catalog reader, or an indexed query on the namespace table.
- The wait-interval check (line 766) and the session-ID parse (line 769)
remain in Go and operate only on the small filtered set.

**Related**

- #171192 — `getApplicableTables` same-shape issue in auto-stats refresher
- Epic CRDB-62272

Jira issue: CRDB-64373

Epic CRDB-58778

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.