cockroachdb / cockroachdb/cockroach

tracing: important bits are dropped from the recorded span with many descendant spans

Open
#102,102 0 comments 0 reactions 0 assignees View on GitHub
A-observability-inf C-bug T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Here is a stmt bundle collected on TPCH Q7 on a single node
[stmt-bundle-859275659091771393.zip](https://github.com/cockroachdb/cockroach/files/11305664/stmt-bundle-859275659091771393.zip)

Here is what we see for `JoinReader/7` on the distsql diagram:
Screenshot 2023-04-23 at 6 50 55 PM

Note that how it has 0 KV bytes read, 0 KV gRPC calls, and 0 MVCC steps which doesn't make sense (given that we did look up a bunch of rows). If we examine the logs, we do see that the streamer performed 48 KV gRPC calls.

Screenshot 2023-04-23 at 6 51 52 PM

I believe the problem is that we cap the recording to have at most 1000 spans
Screenshot 2023-04-23 at 6 52 51 PM

so we end up dropping spans that contained the execution stats metadata.

Andrei did bunch of work in #88414 to improve things, but as this example shows, there is still more improvements that we need to do.

Jira issue: CRDB-27279

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.