cockroachdb / cockroachdb/cockroach
roachprod-microbench: capture database (server) logs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently, we capture stdout/stderr (see `writeBenchmarkErrorLogs`) but not the database logs. E.g.,
```
2025/11/30 22:39:24 report.go:84: Writing error logs for benchmark at BenchmarkVirtualTableQueries-2-timeout-stdout.log, BenchmarkVirtualTableQueries-2-timeout-stderr.log
```
Looking at `BenchmarkVirtualTableQueries-2-timeout-stdout.log`, this was the run which timed out,
```
BenchmarkVirtualTableQueries/show_create_all_triggers
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers3206802361
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers544793610
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers239013880
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers3921498648
test_log_scope.go:82: use -show-logs to present logs inline
```
Note that the database logs are under `/tmp` on the remote instances. Those were not captured. Instead, `BenchmarkVirtualTableQueries-2-timeout-stderr.log` just displays the wrapper command error code,
```
Remote command: sh -c "cd /mnt/data1/0f2bc7b8fb79175f32fe44754a872845cd01e038/pkg/bench/rttanalysis/bin && COCKROACH_RANDOM_SEED=1 && timeout -k 30s 20m ./run.sh -test.short -test.benchmem -test.bench=^BenchmarkVirtualTableQueries$ -test.run=^$ -test.v"
Remote error: COMMAND_PROBLEM: exit status 124
I251130 22:19:24.619795 1 (gostd) rand.go:268 [-] 1 random seed: -5333314734833825258
```
Jira issue: CRDB-57503
Contributor guide
Research direction
Start by locating writeBenchmarkErrorLogs and tracing how roachprod-microbench handles stdout, stderr, and remote instance files. Check how the database logs under /tmp are exposed by the remote command, and consider the work done when timeout reports include those server logs alongside the existing stdout and stderr logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100