cockroachdb / cockroachdb/cockroach
tracing,bulk: investigate missing trace span metadata
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
I'm looking at the per-node trace span summaries dumped as part of the Execution Details on the Advanced Debug job page for a running backup on a 5 node roachprod cluster , and I see something unexpected: some nodes list several expected trace spans and aggregate numbers for them, e.g. node 4 here:
```
# SQL Instance ID: 4 (); Flow/proc ID: f67b5ce8-76c9-4f35-a988-89e856f773d5/0
## Span Totals
- /cockroach.blobs.Blob/PutStream (5): 564.394303ms
- /cockroach.roachpb.Internal/Batch (1018): 11m38.029702802s
- admissionWorkQueueWait (1814): 12m30.055944678s
- backup.ExportRequest (982): 11m38.344886304s
- backup.FileSSTSink.Write (981): 3m27.316809012s
- backup.worker (5): 15m6.847412363s
- backupDataProcessor (1): 3m8.249150213s
- backupDataProcessor.runBackupProcessor (1): 3m8.248739347s
- dist sender send (1014): 11m38.361106861s
- evalExport (983): 1m19.609223805s
- exportRequestLimiter (47): 2.782670928s
- noop (1): 3m8.249630987s
- range lookup (1): 461.479µs
- storage.MVCCExportToSST (983): 1m19.597905724s
- txn coordinator send (31): 36.077011ms
## Aggregate Stats
- cockroach.ccl.backupccl.ExportStats:
num_files: 981
data_size: 3555.90 MB
throughput: 18.92 MB/s
- cockroach.util.admission.admissionpb.AdmissionWorkQueueStats:
queue (kv-elastic-cpu-queue/bulk-normal-pri) wait: 12m30s
```
But for node 5, we don't see many/most of those spans:
```
# SQL Instance ID: 5 (); Flow/proc ID: f67b5ce8-76c9-4f35-a988-89e856f773d5/0
## Span Totals
- /cockroach.sql.distsqlrun.DistSQL/FlowStream (1): 3m13.486754275s
- flowinfra.RemoteFlowRunner: waiting for flow to finish (1): 3m13.486799737s
- outbox (1): 3m13.486768656s
- watchdog (1): 3m13.486700899s
## Aggregate Stats
- cockroach.ccl.backupccl.ExportStats:
num_files: 1060
data_size: 3821.93 MB
throughput: 19.78 MB/s
```
We do see _some_ spans and we see stats for node 5, so it is in the flow/did send metas that got added to the aggregation -- so where are the other spans? there should be a backup processor span at least... right?
To reproduce:
```
roachprod create -n 5 $CLUSTER
roachprod {stage, start} $CLUSTER
roachprod run $CLUSTER:1 -- ./cockroach workload fixtures import tpcc --warehouses=1000
roachprod sql $CLUSTER:1 -- -e "backup into 'nodelocal://1/a' with detached"
roachprod adminurl $CLUSTER:1 --open
```
And browse to the backup job page's Advanced Debugging tab, wait ~30s for trace data to accumulate to the coordinator and then request execution details and view the by-node summary.
Jira issue: CRDB-49162
Contributor guide
Assessment
This issue has not been assessed yet.