cockroachdb / cockroachdb/cockroach

Allow tsdump to create tsdump.yaml mapping file automatically when creating the tsdump

Open
#113,937 2 comments 0 reactions 0 assignees View on GitHub
A-observability-inf C-enhancement T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### **Is your feature request related to a problem? Please describe.**

In order to load tsdump raw files to be able to review the time series data, [we need to also generate the tsdump.yaml mappings. ](https://www.cockroachlabs.com/docs/stable/cockroach-debug-tsdump) This as it currently has been designed is cumbersome, since we also need to either obtain the data from the cli by asking the customer to run a sql query, or request the debug zip, then download it, which can be rather time consuming since debug zips can be fairly large.

### **Describe the solution you'd like**

The ideal scenario would be for the tsdump.yaml mapping file to be automatically generated at the same time as the customer is creating the tsdump in raw format (this is the format required by TSEs to perform investigations and this is the only format we need the yaml file for). I have modified our source code to do exactly this as to prove this is possible.

Usage: If the customer choses `--format=raw`, the `tsdump.yaml` is automatically created in `/tmp`, however, the customer can pass an additional parameter (`--yaml`) and provide the path they would like to have the yaml file saved into, as shown in the examples below.

Please notice, the `yaml` file is only created when `--format=raw` is used.

There are 2 files that need to be modified:

1) [pkg/cli/tsdump.go](pkg/cli/tsdump.go)

`Added the following additional import`

```
"github.com/cockroachdb/cockroach/pkg/cli/clisqlclient" // Added by Daniel Almeida to enable yaml to be created
```

`Modified this section by adding the yaml option`

```
var debugTimeSeriesDumpOpts = struct {
format tsDumpFormat
from, to timestampValue
clusterLabel string
yaml string
}{
format: tsDumpText,
from: timestampValue{},
to: timestampValue(timeutil.Now().Add(24 * time.Hour)),
clusterLabel: "",
yaml: "/tmp/tsdump.yaml", // If a yaml file isn't passed as an argument, we default to /tmp/tsdump.yaml. This needs to be defined in cli/debug.go (line 1546)
}
```

`Added the following call to our custom function to generate the yaml to the debugTimeSeriesDumpCmd command`, snippet of the affected are below

```
// Daniel Almeida added the function below to create tsdump.yaml when creating tsdump
generateTsdumpYaml()
```

Sample below for clarity, so you can see where the above was added:

```
var debugTimeSeriesDumpCmd = &cobra.Command{
Use: "tsdump",
Short: "dump all the raw timeseries values in a cluster",
Long: `
Dumps all of the raw timeseries values in a cluster. Only the default resolution
is retrieved, i.e. typically datapoints older than the value of the
'timeseries.storage.resolution_10s.ttl' cluster setting will be absent from the
output.

When an input file is provided instead (as an argument), this input file
must previously have been created with the --format=raw switch. The command
will then convert it to the --format requested in the current invocation.
`,
Args: cobra.RangeArgs(0, 1),
RunE: clierrorplus.MaybeDecorateError(func(cmd *cobra.Command, args []string) error {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

var convertFile string
if len(args) > 0 {
convertFile = args[0]
}

var w tsWriter
switch debugTimeSeriesDumpOpts.format {
case tsDumpRaw:
if convertFile != "" {
return errors.Errorf("input file is already in raw format")
}
// Daniel Almeida added the function below to create tsdump.yaml when creating tsdump in raw format
generateTsdumpYaml()
// Special case, we don't go through the text output code.
case tsDumpCSV:
w = csvTSWriter{w: csv.NewWriter(os.Stdout)}
case tsDumpTSV:
cw := csvTSWriter{w: csv.NewWriter(os.Stdout)}
cw.w.Comma = '\t'
w = cw
case tsDumpText:
w = defaultTSWriter{w: os.Stdout}
case tsDumpOpenMetrics:
w = makeOpenMetricsWriter(os.Stdout)
default:
return errors.Newf("unknown output format: %v", debugTimeSeriesDumpOpts.format)
}
...
.....
....... removed the rest of code for brevity.
```

`Created function the generate the yaml: generateTsdumpYaml`

```
// Added by Daniel Almeida to obtain node to store mappings when creating tsdump so we automatically create the tsdump.yaml
func generateTsdumpYaml() (resErr error) {
file, err := os.OpenFile(debugTimeSeriesDumpOpts.yaml, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
return err
}

ctx := context.Background()
sqlConn, err := makeSQLClient(ctx, "tsdump-node-to-store-mapping", useSystemDb)
if err != nil {
return err
}
defer func() { resErr = errors.CombineErrors(resErr, sqlConn.Close()) }()

_, rows, err := sqlExecCtx.RunQuery(
ctx,
sqlConn,
clisqlclient.MakeQuery(`SELECT store_id || ': ' || node_id FROM crdb_internal.kv_store_status`), false)

if err != nil {
return err
}

var strStoreNodeID string
for i := range rows {
storeNodeID := rows[i]
// This is where the magic happens and we print the storeID: nodeID mappings to the default /tmp/tsdump.yaml if a destination is not provided
strStoreNodeID = strings.Join(storeNodeID, " ")
strStoreNodeID += "\n"
file.WriteString(strStoreNodeID)
}
return
}
```

2) Edited the file [pkg/cli/debug.go](pkg/cli/debug.go) and added the `debugTimeSeriesDumpOpts.yaml` argument to the `f = debugTimeSeriesDumpCmd.Flags()` section, as shown below:

```
f = debugTimeSeriesDumpCmd.Flags()
f.Var(&debugTimeSeriesDumpOpts.format, "format", "output format (text, csv, tsv, raw, openmetrics)")
f.Var(&debugTimeSeriesDumpOpts.from, "from", "oldest timestamp to include (inclusive)")
f.Var(&debugTimeSeriesDumpOpts.to, "to", "newest timestamp to include (inclusive)")
f.StringVar(&debugTimeSeriesDumpOpts.clusterLabel, "cluster-label",
"", "prometheus label for cluster name")
f.StringVar(&debugTimeSeriesDumpOpts.yaml, "yaml", debugTimeSeriesDumpOpts.yaml, "path to tsdump.yaml with nodeid-storeid mappings")
```

### **Describe alternatives you've considered**

I compiled the above changes and verified this works as expected. Example usage below:

1) Creating a tsdump, without passing any extra arguments, creates a yaml file by default in `/tmp/tsdump.yaml`

```
❯ ./cockroach debug tsdump --insecure --host lab-kub01:30007 --format raw --from='2023-11-06 22:00:00' --to='2023-11-06 23:59:59' > /Users/daniel/Documents/customerDebug/lab/tsdump.gob
```

```
❯ cat /tmp/tsdump.yaml
1: 1
2: 2
3: 3
```

2) Creating a tsdump with a custom path for the yaml `tsdump.yaml` :

```
❯ ./cockroach debug tsdump --insecure --host lab-kub01:30007 --format raw --from='2023-11-06 22:00:00' --to='2023-11-06 23:59:59' --yaml=/Users/daniel/Documents/customerDebug/lab/tsdump.yaml > /Users/daniel/Documents/customerDebug/lab/tsdump.gob
```

```
❯ ls -alh /Users/daniel/Documents/customerDebug/lab
total 22664
drwxr-xr-x 4 daniel staff 128B Nov 7 00:05 .
drwxr-xr-x@ 37 daniel staff 1.2K Nov 6 12:20 ..
-rw-r--r--@ 1 daniel staff 10M Nov 7 00:05 tsdump.gob
-rw-r--r--@ 1 daniel staff 15B Nov 7 00:05 tsdump.yaml
```

```
❯ cat /Users/daniel/Documents/customerDebug/lab/tsdump.yaml
1: 1
2: 2
3: 3
```

### **Additional context**

This would save a lot of time for customers and TSEs when working with time series data, as it would eliminate the requirement to having a debug zip in order to figure out the node to store mappings.

Jira issue: CRDB-33265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.