cockroachdb / cockroachdb/cockroach

drtprod, roachprod: improve logging noise ratio

Open
#150,411 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra A-testing autosolve C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently it's hard to debug or review logs produced by `drtprod` & `roachprod` when creating large clusters.

As an example from a recent 300-node cluster creation:

```
Jul 17 08:54:52 drt-scale-300-bench-monitor-0001 bash[90904]: 2025/07/17
08:54:52 roachprod.go:891: creating cluster config failed for the ip:ports
234:[34.63.180.126:9100,34.63.180.126:9435,34.63.180.126:26258,34.63.180.126:2112,34.63.180.126:2113,
34.63.180.126:2114,34.63.180.126:2115,34.63.180.126:2116,34.63.180.126:2117,
34.63.180.126:2118,34.63.180.126:2119,34.63.180.126:2120]
273:[34.10.229.108:9100,34.10.229.108:9435,34.10.229.108:26258,
34.10.229.108:2112,34.10.229.108:2113,34.10.229.108:2114,34.10.229.108:2115,
34.10.229.108:2116,34.10.229.108:2117,34.10.229.108:2118,
...
```

This just a small snippet that has been truncated where all the IPs in the cluster are printed out.

Other things we might consider is adding the node a command is executing for to the log output to make it a bit easier to follow.
There might be other ways to make the log more readable that we should think about in terms of large clusters.

There are also no log levels, or verbosity levels in `roachprod`, so it's not easy to find errors, warning specifically. These are mostly hardcoded into the log output, sometimes with `WARN:` etc.

Jira issue: CRDB-52701

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.