cockroachdb / cockroachdb/cockroach
jobs: enable tracing by default and persist traces for all job executions
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Job executions are low-cardinality and not as performance-sensitive as SQL queries, yet they have significantly less tracing infrastructure. Debugging job failures and performance issues today is difficult because:
- Only jobs implementing the `TraceableJob` interface with `ForceRealSpan()=true` get real tracing spans (import, schema changes, backup, restore, logical replication, stream ingestion). All other job types get noop spans that discard recordings.
- Even for traced jobs, traces are **ephemeral** — they exist only in-memory and are only queryable via `crdb_internal.cluster_inflight_traces` while the job is running. Once the job completes, the trace is gone.
- `cockroach debug job-trace` only works for in-flight jobs, making post-mortem debugging impossible.
- The `DumpTraceAfterRun()` mechanism can persist traces to `system.job_info`, but it's opt-in per job type. Persisted traces are only accessible via the DB Console's Advanced Debugging tab or raw HTTP API — there's no SQL-level access.
**Current state of the code:**
- **Span creation**: [`pkg/jobs/adopt.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/jobs/adopt.go) creates root spans for jobs, but defaults to noop unless `ForceRealSpan()=true`
- **Trace ID persistence**: The trace ID is stored in `jobspb.Progress.TraceID` and `system.job_message` (kind='trace-id')
- **Trace dump**: `maybeDumpTrace()` in `adopt.go` persists traces to `system.job_info` as gzip-compressed chunked protobuf (`jobspb.TraceData`), but only for jobs with `DumpTraceAfterRun()=true`
- **Reading traces**: Available via `/_status/job_profiler_execution_details` HTTP API or DB Console Advanced Debugging tab. No SQL interface.
- **Jaeger conversion**: `Recording.ToJaegerJSON()` already exists in [`pkg/util/tracing/tracingpb/recording.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/tracing/tracingpb/recording.go)
- **OTLP export**: Real-time OTLP export exists via `trace.opentelemetry.collector` cluster setting, but only for live spans
**Describe the solution you'd like**
Enable tracing for all jobs by default and persist traces with a retention policy. Suggested implementation plan:
1. **Make all jobs traceable by default**
- Remove the `ForceRealSpan()` opt-in gate in `pkg/jobs/adopt.go` — always create real spans with structured recording for all job types
- Keep the `TraceableJob` interface for jobs that want verbose recording or custom behavior
2. **Always dump traces on job completion**
- Remove the `DumpTraceAfterRun()` opt-in gate — always call `maybeDumpTrace()` after job completion (success or failure)
- Add a cluster setting (e.g., `jobs.trace.enabled`, default `true`) to allow disabling if needed
3. **Add trace retention/GC**
- Add a cluster setting `jobs.trace.retention_period` (default 7 days or similar)
- Implement GC that prunes old trace data from `system.job_info` entries matching the `~profiler/*/resumer-trace/*` pattern
- Could piggyback on existing job GC infrastructure
4. **Surface traces via SQL**
- Add `SHOW JOB TRACE ` syntax that reads persisted `TraceData` from `system.job_info` and renders human-readable text
- Optionally support Jaeger JSON output (using existing `ToJaegerJSON()`)
- Or expose via a `crdb_internal.job_traces` virtual table
5. **Optional: OTLP export for completed job traces**
- When `trace.opentelemetry.collector` is configured, push completed job traces to the OTLP collector on job completion
- This would allow job traces to appear in external observability tools (Jaeger, Datadog, etc.) alongside other CRDB traces
**Describe alternatives you've considered**
- **Status quo with better docs**: The existing `DumpTraceAfterRun()` opt-in could be extended to more job types incrementally, but this doesn't address the discoverability problem or provide SQL access to traces.
- **External-only approach**: Rely entirely on OTLP export to external systems. This avoids storing traces in `system.job_info` but requires external infrastructure and doesn't help with `cockroach debug job-trace` or SQL-level debugging.
**Additional context**
The SQL query tracing story provides a useful comparison. SQL queries use a multi-tier approach: sampled stats aggregated in `system.statement_statistics`, on-demand verbose tracing via `SET TRACING` or statement diagnostics, and full bundle collection via `EXPLAIN ANALYZE (DEBUG)`. Jobs are lower cardinality than SQL queries and individually more important for debugging, yet they currently have less tracing infrastructure. Enabling tracing by default for jobs would close this gap with minimal performance impact.
Epic: none
Jira issue: CRDB-62733
Contributor guide
Assessment
This issue has not been assessed yet.