cockroachdb / cockroachdb/cockroach

sql/insights: serve the DB Console insights page from a dedicated API over persisted views

Open
#172,492 1 comment 0 reactions 0 assignees View on GitHub
A-sql-console-general A-sql-observability C-enhancement O-agent T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

The DB Console workload insights page (statement + transaction insights)
fetches data by running raw SQL over the generic sql-over-http endpoint
(`api/v2/sql/`, via `executeInternalSql`) against the
`crdb_internal.cluster_execution_insights` and
`crdb_internal.cluster_txn_execution_insights` views. This has several
problems:

- **RPC fan-out on every load.** Those views fan out to every node and
dedup at the view layer — slow on large clusters (the same cost called out
in #171329).
- **No server-side pagination.** The queries carry no `LIMIT`/`OFFSET`; the
page pulls the full deduped result set and does filtering, sorting, and
pagination client-side (`filterStatementInsights` /
`filterTransactionInsights` + `SortedTable`). That was acceptable against
the bounded in-memory FIFO buffer, but does not scale once insights are
persisted to a system table (#171329) whose size is governed by
`sql.insights.persisted_rows.max`.
- **Coupling to an unstable surface.** The frontend depends on the unstable
`crdb_internal` schema and on brittle inline SQL strings.

**Describe the solution you'd like**

Once #171329 (persistence) and #171330 (`information_schema` views) land,
back the insights page with a dedicated read-only HTTP API under `/api/v2/`
that reads from the new `information_schema.crdb_statement_execution_insights`
/ `crdb_transaction_execution_insights` views, replacing the sql-over-http
calls.

Mirror the database/table-metadata precedent, which already moved a DB
Console observability surface off sql-over-http:

- Server handler(s) registered in `pkg/server/api_v2.go` `routeDefinitions`
(e.g. `sql_insights/`, or split `statement_insights/` +
`transaction_insights/`), returning JSON and doing filtering, sorting, and
pagination server-side. See `pkg/server/api_v2_databases_metadata.go`.
- The predicates currently embedded in the SQL — time range, internal-app
exclusion, `problem(s) != 'None'`, and fingerprint dedup — move into the
endpoint, alongside the currently client-side app-name filter, sort, and
page params as query parameters.
- Client API module using `fetchDataJSON` + an SWR hook
(`useSwrWithClusterId`), replacing the `executeInternalSql` calls. See
`pkg/ui/workspaces/cluster-ui/src/api/databases/getDatabaseMetadataApi.ts`.

Endpoints needed to cover the page:

- statement insights list (overview)
- transaction insights list (overview)
- transaction insight details (txn + its statements) — replaces
`getTxnInsightDetailsApi`
- statement insight details / by fingerprint

Same `VIEWACTIVITY` / `VIEWACTIVITYREDACTED` privilege model as today.

**Known tradeoff: freshness**

The persisted `information_schema` views only contain *flushed* data, so the
page will not show insights newer than the last flush
(`sql.insights.flush.interval` + jitter). This is deliberate: #171329 keeps
the `crdb_internal` RPC path available for querying fresh in-memory insights,
so we accept the flush-interval lag on the DB Console page in exchange for
dropping the fan-out and scaling to the persisted table. If sub-flush
recency later proves necessary, a follow-up could union fresh in-memory +
persisted results.

**Describe alternatives you've considered**

- **Repoint the existing queries only** (swap the `FROM` clause to the
`information_schema` views, keep sql-over-http). Gets off
`crdb_internal`/fan-out with a ~2-file change, but keeps the generic
transport, the brittle inline SQL, and — critically — client-side
pagination that does not scale to the persisted table's row count.
- **Keep querying `crdb_internal.cluster_execution_insights`.** Unstable
schema, and pays the RPC fan-out even for already-flushed data.

**Additional context**

- Depends on #171329 (persist) and #171330 (`information_schema` views);
blocked until both land.
- Reference implementation to mirror: database/table metadata endpoints —
[`pkg/server/api_v2_databases_metadata.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/server/api_v2_databases_metadata.go)
and client
[`getDatabaseMetadataApi.ts`](https://github.com/cockroachdb/cockroach/blob/master/pkg/ui/workspaces/cluster-ui/src/api/databases/getDatabaseMetadataApi.ts).
Route table:
[`pkg/server/api_v2.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/server/api_v2.go).
- Current insights fetch code to migrate:
[`stmtInsightsApi.ts`](https://github.com/cockroachdb/cockroach/blob/master/pkg/ui/workspaces/cluster-ui/src/api/stmtInsightsApi.ts),
[`txnInsightsApi.ts`](https://github.com/cockroachdb/cockroach/blob/master/pkg/ui/workspaces/cluster-ui/src/api/txnInsightsApi.ts),
[`txnInsightDetailsApi.ts`](https://github.com/cockroachdb/cockroach/blob/master/pkg/ui/workspaces/cluster-ui/src/api/txnInsightDetailsApi.ts)
(all use `executeInternalSql` -> `api/v2/sql/`). The insights fetch layer
already uses SWR hooks; page filters/sort/time-scale still use Redux.

Epic CRDB-65260

Jira issue: CRDB-65736

Contributor guide

Open the contributing guide

Research direction

Start with pkg/server/api_v2_databases_metadata.go and api_v2.go, then inspect stmtInsightsApi.ts, txnInsightsApi.ts, and txnInsightDetailsApi.ts alongside the persistence and information_schema prerequisites. Done means dedicated v2 endpoints provide filtered, sorted, paginated statement and transaction insights plus details, and the client uses fetchDataJSON with SWR instead of executeInternalSql.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql, typescript
Domain
backend-api-design, databases, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.