cockroachdb / cockroachdb/cockroach

TSdump: Reveal all tsdump.gob.yaml errors when file can't load

Open
#95,437 0 comments 0 reactions 0 assignees View on GitHub
A-observability-inf C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

There are a number of related issues when attempting to stage a tsdump regarding the yaml file when it doesn't have all of the stores/nodes that it needs, and when it has stores/nodes that there is no timeseries data for.

This occurs when the tsdump is taken at one point, and the debug zip from which we generate the tsdump.gob.yaml file is taken at another point when nodes/stores have been added/removed

**More stores/nodes than needed**

This issue occurs when there are stores/nodes in the yaml file for which there is no timeseries data. In this case, you will only find out this is the case after attempting to import the tsdump. The command will attempt to import all data, get to the very end, and supply an error like the below:

```
E230118 15:26:27.581515 1 1@cli/clierror/check.go:35 ⋮ [-] 492 ‹ERROR›: cockroach server exited with error: consider updating the mapping file ‹/mnt/data1/tsdump.gob.yaml› or restarting the server with COCKROACH_DEBUG_TS_IMPORT_FILE=- to ignore the error: s9 supplied in input mapping, but no timeseries found for it
```

There are two problems with this:

1. If there are other stores supplied in the mapping input that there is no timeseries data for in the tsdump, it will not expose those stores. In the above example, I removed `s9:n9` from the yaml file, re-ran the tsdump import, waited 15 minutes for the data to import, and then ran into the same error again but for `s10`. Then again for `s11`.
2. Arguably, there is no need to error out in this situation. The tsdump could output a warning for each supplied store for which there is no timeseries data, and then allow the data to be staged anyway instead of going into failure mode

The above error should be outputted at the beginning of the loading of the time series data and output all stores for which it doesn't have data before going into failure mode, or not go into failure mode at all and simply display a warning (if possible).

**Fewer stores/nodes than needed**

This issue occurs when there is timeseries data for which there are no stores/nodes associated:

```
* ERROR: ERROR: cockroach server exited with error: consider updating the mapping file tsdump.gob.yaml or restarting the server with COCKROACH_DEBUG_TS_IMPORT_FILE=- to ignore the error: need to map the remaining stores map[2:{} 3:{}] to nodes
*
ERROR: cockroach server exited with error: consider updating the mapping file tsdump.gob.yaml or restarting the server with COCKROACH_DEBUG_TS_IMPORT_FILE=- to ignore the error: need to map the remaining stores map[2:{} 3:{}] to nodes
Failed running "start-single-node"
```

Again, there may be a way to just output a warning and not error out completely - just stage the data that's there and let us know that there may be data missing.

-----------

If the tsdump also went out to `kv_store_status` and generated the yaml file itself, all of the above could be avoided as well. It would also remove the dependency of needing a debug zip to generate the yaml file off of, or the dependency of asking the customer to send the output of `select * from crdb_internal.kv_store_status;`.

Jira issue: CRDB-23516

Epic CRDB-31306

Contributor guide

Open the contributing guide

Research direction

Start by tracing the tsdump staging/import flow that loads tsdump.gob.yaml and reports missing or extra stores and nodes. Compare the mapping data with the timeseries data and the proposed kv_store_status source; done should mean all relevant errors are revealed together, or the available data is staged with clear warnings. No specific source files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.