cockroachdb / cockroachdb/cockroach
jobs,bulk: job trace "Span Totals" omits coordinator-side spans
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Clicking through to a job's trace info for a RESTORE, we went looking for the timing of a step that runs on the job coordinator before the DistSQL flow is planned, and it wasn't there — even though the span exists, is created off the job's context, and shows up correctly in the job's own recording (crdb_internal.cluster_inflight_traces renders it as [<span name>: {count: 1, duration 22.2s}]). The reason is that the "Span Totals" section on that page is not built from the job's span recording at all: it comes from FlushTracingAggregatorStats in aggregator_stats.go, which aggregates SpanTotals from per-node tracing aggregators keyed by execinfrapb.ComponentID — i.e. per DistSQL flow/processor — so only spans created inside a flow component are ever summarized there. Anything the coordinator does outside the flow (planning, manifest loading, pre-flow setup work) is structurally invisible on that page, which is misleading because the page reads as "the job's trace" and coordinator work is often exactly what you're trying to account for. Options, roughly in increasing order of effort: document that the section is flow-only and point at the job trace dump for coordinator spans; have the job resumer contribute its own recording's childrenMetadata into the same summary so coordinator spans appear alongside flow components; or give the coordinator a synthetic component ID with a tracing aggregator so its spans flow through the existing path unchanged.
Jira issue: CRDB-68562
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/util/bulk/aggregator_stats.go and the FlushTracingAggregatorStats entry point to trace how SpanTotals are collected from per-node tracing aggregators. Compare that flow with the job recording and its childrenMetadata, then agree on whether the fix is documentation or aggregation changes; done means coordinator-side spans are either clearly accounted for or explicitly directed to the job trace dump.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100