cockroachdb / cockroachdb/cockroach

roachprod-microbench: capture database (server) logs

Open
#158,821 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, we capture stdout/stderr (see `writeBenchmarkErrorLogs`) but not the database logs. E.g.,

```
2025/11/30 22:39:24 report.go:84: Writing error logs for benchmark at BenchmarkVirtualTableQueries-2-timeout-stdout.log, BenchmarkVirtualTableQueries-2-timeout-stderr.log
```

Looking at `BenchmarkVirtualTableQueries-2-timeout-stdout.log`, this was the run which timed out,

```
BenchmarkVirtualTableQueries/show_create_all_triggers
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers3206802361
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers544793610
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers239013880
test_log_scope.go:82: use -show-logs to present logs inline
testing_shims.go:40: -- test log scope end --
test_log_scope.go:171: test logs captured to: /tmp/logBenchmarkVirtualTableQueries_show_create_all_triggers3921498648
test_log_scope.go:82: use -show-logs to present logs inline
```

Note that the database logs are under `/tmp` on the remote instances. Those were not captured. Instead, `BenchmarkVirtualTableQueries-2-timeout-stderr.log` just displays the wrapper command error code,

```
Remote command: sh -c "cd /mnt/data1/0f2bc7b8fb79175f32fe44754a872845cd01e038/pkg/bench/rttanalysis/bin && COCKROACH_RANDOM_SEED=1 && timeout -k 30s 20m ./run.sh -test.short -test.benchmem -test.bench=^BenchmarkVirtualTableQueries$ -test.run=^$ -test.v"
Remote error: COMMAND_PROBLEM: exit status 124
I251130 22:19:24.619795 1 (gostd) rand.go:268 [-] 1 random seed: -5333314734833825258
```

Jira issue: CRDB-57503

Contributor guide

Open the contributing guide

Research direction

Start by locating writeBenchmarkErrorLogs and tracing how roachprod-microbench handles stdout, stderr, and remote instance files. Check how the database logs under /tmp are exposed by the remote command, and consider the work done when timeout reports include those server logs alongside the existing stdout and stderr logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.