cockroachdb / cockroachdb/cockroach
sql/stats: auto-stats refresher lacks observability for cold-start, queue depth, and GC
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary**
The auto-stats refresher (`pkg/sql/stats/automatic_stats.go`) emits essentially
no logging or metrics for the codepaths that are expected to be load-bearing at
high table counts (10K+). This makes it impossible to characterize cold-start
and steady-state behavior of the subsystem from a running cluster without
modifying the binary.
Surfaced while planning the work in CRDB-63468 (validate auto-stats refresher
at 100K tables). The lack of observability is the first blocker — we can't
answer the questions the test plan asks without it.
**What's missing**
1. **`getApplicableTables` / `ensureAllTables` (cold start).** At every node
start, the refresher scans all descriptors in a single transaction
(`txn.Descriptors().GetAll(...)`) to seed `mutationCounts`. At 100K tables
this is a multi-GB read; at 1M tables it dominates startup. Today this
function logs only on error — no start, no completion, no duration, no
descriptor count, no transaction stats.
2. **Refresh tick (steady state).** The per-minute refresh loop iterates over
`mutationCounts` from a single goroutine. There is no log or metric for:
- `len(mutationCounts)` at the top of each tick (queue depth)
- per-tick wall-clock duration
- number of tables processed before the next tick fires
- number of `maybeRefreshStats` invocations that resulted in a job
3. **Stats GC.** `deleteStatsForDroppedTables` runs hourly and is bounded at
1000 rows per pass. It logs only on error — no row count, no duration. We
cannot tell whether GC is keeping pace with table drops just by reading
logs.
**Code references**
- [pkg/sql/stats/automatic_stats.go#L871](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L871) — `getApplicableTables`, logs only on error
- [pkg/sql/stats/automatic_stats.go#L633](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L633) — `ensureAllTables` call site
- [pkg/sql/stats/automatic_stats.go#L627-L770](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L627-L770) — refresh loop with no telemetry
- [pkg/sql/stats/automatic_stats.go#L840-L867](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L840-L867) — stats GC task
A grep of the file finds no `metric.Counter`, `metric.Gauge`, or
`metric.Histogram` registrations.
**Suggested minimum addition**
- `log.Infof` at start and end of `getApplicableTables` with elapsed time and
number of tables seeded
- `log.Infof` at the top of each refresh tick with `len(mutationCounts)` and
per-tick processing duration
- `log.Infof` in stats GC with rows deleted and elapsed time
- Histograms / counters: `sql.stats.refresher.startup_duration`,
`sql.stats.refresher.queue_depth`, `sql.stats.refresher.tick_duration`,
`sql.stats.gc.rows_deleted`
Even just the log lines would unblock CRDB-63468. Metrics are a follow-up.
**Workarounds available to testers today**
- Long-running transaction view in DB Console to infer cold-start duration
- Per-node RSS in Datadog / Grafana to infer memory peak
- Goroutine dumps (`/debug/pprof/goroutine?debug=2`) to see the single-goroutine
fanout pattern and infer queue depth
- `SELECT count(*) FROM system.jobs WHERE ...` sampling for throughput
These are all proxies. None give the precision the test plan calls for.
Epic CRDB-58778
Jira issue: CRDB-64360
Contributor guide
Assessment
This issue has not been assessed yet.