cockroachdb / cockroachdb/cockroach

sql/stats: auto-stats refresher lacks observability for cold-start, queue depth, and GC

Open
#171,184 1 comment 0 reactions 1 assignee Claimed by @andyyang890 View on GitHub
A-many-descriptors A-sql-logging-and-telemetry A-sql-table-stats C-bug C-enhancement E-quick-win O-agent P-2 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary**

The auto-stats refresher (`pkg/sql/stats/automatic_stats.go`) emits essentially
no logging or metrics for the codepaths that are expected to be load-bearing at
high table counts (10K+). This makes it impossible to characterize cold-start
and steady-state behavior of the subsystem from a running cluster without
modifying the binary.

Surfaced while planning the work in CRDB-63468 (validate auto-stats refresher
at 100K tables). The lack of observability is the first blocker — we can't
answer the questions the test plan asks without it.

**What's missing**

1. **`getApplicableTables` / `ensureAllTables` (cold start).** At every node
start, the refresher scans all descriptors in a single transaction
(`txn.Descriptors().GetAll(...)`) to seed `mutationCounts`. At 100K tables
this is a multi-GB read; at 1M tables it dominates startup. Today this
function logs only on error — no start, no completion, no duration, no
descriptor count, no transaction stats.

2. **Refresh tick (steady state).** The per-minute refresh loop iterates over
`mutationCounts` from a single goroutine. There is no log or metric for:
- `len(mutationCounts)` at the top of each tick (queue depth)
- per-tick wall-clock duration
- number of tables processed before the next tick fires
- number of `maybeRefreshStats` invocations that resulted in a job

3. **Stats GC.** `deleteStatsForDroppedTables` runs hourly and is bounded at
1000 rows per pass. It logs only on error — no row count, no duration. We
cannot tell whether GC is keeping pace with table drops just by reading
logs.

**Code references**

- [pkg/sql/stats/automatic_stats.go#L871](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L871) — `getApplicableTables`, logs only on error
- [pkg/sql/stats/automatic_stats.go#L633](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L633) — `ensureAllTables` call site
- [pkg/sql/stats/automatic_stats.go#L627-L770](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L627-L770) — refresh loop with no telemetry
- [pkg/sql/stats/automatic_stats.go#L840-L867](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stats/automatic_stats.go#L840-L867) — stats GC task

A grep of the file finds no `metric.Counter`, `metric.Gauge`, or
`metric.Histogram` registrations.

**Suggested minimum addition**

- `log.Infof` at start and end of `getApplicableTables` with elapsed time and
number of tables seeded
- `log.Infof` at the top of each refresh tick with `len(mutationCounts)` and
per-tick processing duration
- `log.Infof` in stats GC with rows deleted and elapsed time
- Histograms / counters: `sql.stats.refresher.startup_duration`,
`sql.stats.refresher.queue_depth`, `sql.stats.refresher.tick_duration`,
`sql.stats.gc.rows_deleted`

Even just the log lines would unblock CRDB-63468. Metrics are a follow-up.

**Workarounds available to testers today**

- Long-running transaction view in DB Console to infer cold-start duration
- Per-node RSS in Datadog / Grafana to infer memory peak
- Goroutine dumps (`/debug/pprof/goroutine?debug=2`) to see the single-goroutine
fanout pattern and infer queue depth
- `SELECT count(*) FROM system.jobs WHERE ...` sampling for throughput

These are all proxies. None give the precision the test plan calls for.

Epic CRDB-58778
Jira issue: CRDB-64360

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.