cockroachdb / cockroachdb/cockroach

tracing,bulk: investigate missing trace span metadata

Open
#143,920 3 comments 0 reactions 1 assignee Claimed by @kev-cao View on GitHub
A-disaster-recovery C-investigation P-1 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

I'm looking at the per-node trace span summaries dumped as part of the Execution Details on the Advanced Debug job page for a running backup on a 5 node roachprod cluster , and I see something unexpected: some nodes list several expected trace spans and aggregate numbers for them, e.g. node 4 here:

```
# SQL Instance ID: 4 (); Flow/proc ID: f67b5ce8-76c9-4f35-a988-89e856f773d5/0

## Span Totals

- /cockroach.blobs.Blob/PutStream (5): 564.394303ms
- /cockroach.roachpb.Internal/Batch (1018): 11m38.029702802s
- admissionWorkQueueWait (1814): 12m30.055944678s
- backup.ExportRequest (982): 11m38.344886304s
- backup.FileSSTSink.Write (981): 3m27.316809012s
- backup.worker (5): 15m6.847412363s
- backupDataProcessor (1): 3m8.249150213s
- backupDataProcessor.runBackupProcessor (1): 3m8.248739347s
- dist sender send (1014): 11m38.361106861s
- evalExport (983): 1m19.609223805s
- exportRequestLimiter (47): 2.782670928s
- noop (1): 3m8.249630987s
- range lookup (1): 461.479µs
- storage.MVCCExportToSST (983): 1m19.597905724s
- txn coordinator send (31): 36.077011ms

## Aggregate Stats

- cockroach.ccl.backupccl.ExportStats:
num_files: 981
data_size: 3555.90 MB
throughput: 18.92 MB/s

- cockroach.util.admission.admissionpb.AdmissionWorkQueueStats:
queue (kv-elastic-cpu-queue/bulk-normal-pri) wait: 12m30s
```

But for node 5, we don't see many/most of those spans:
```
# SQL Instance ID: 5 (); Flow/proc ID: f67b5ce8-76c9-4f35-a988-89e856f773d5/0

## Span Totals

- /cockroach.sql.distsqlrun.DistSQL/FlowStream (1): 3m13.486754275s
- flowinfra.RemoteFlowRunner: waiting for flow to finish (1): 3m13.486799737s
- outbox (1): 3m13.486768656s
- watchdog (1): 3m13.486700899s

## Aggregate Stats

- cockroach.ccl.backupccl.ExportStats:
num_files: 1060
data_size: 3821.93 MB
throughput: 19.78 MB/s
```

We do see _some_ spans and we see stats for node 5, so it is in the flow/did send metas that got added to the aggregation -- so where are the other spans? there should be a backup processor span at least... right?

To reproduce:
```
roachprod create -n 5 $CLUSTER
roachprod {stage, start} $CLUSTER
roachprod run $CLUSTER:1 -- ./cockroach workload fixtures import tpcc --warehouses=1000
roachprod sql $CLUSTER:1 -- -e "backup into 'nodelocal://1/a' with detached"
roachprod adminurl $CLUSTER:1 --open
```
And browse to the backup job page's Advanced Debugging tab, wait ~30s for trace data to accumulate to the coordinator and then request execution details and view the by-node summary.

Jira issue: CRDB-49162

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.