cockroachdb / cockroachdb/cockroach
drtprod, roachprod: improve logging noise ratio
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently it's hard to debug or review logs produced by `drtprod` & `roachprod` when creating large clusters.
As an example from a recent 300-node cluster creation:
```
Jul 17 08:54:52 drt-scale-300-bench-monitor-0001 bash[90904]: 2025/07/17
08:54:52 roachprod.go:891: creating cluster config failed for the ip:ports
234:[34.63.180.126:9100,34.63.180.126:9435,34.63.180.126:26258,34.63.180.126:2112,34.63.180.126:2113,
34.63.180.126:2114,34.63.180.126:2115,34.63.180.126:2116,34.63.180.126:2117,
34.63.180.126:2118,34.63.180.126:2119,34.63.180.126:2120]
273:[34.10.229.108:9100,34.10.229.108:9435,34.10.229.108:26258,
34.10.229.108:2112,34.10.229.108:2113,34.10.229.108:2114,34.10.229.108:2115,
34.10.229.108:2116,34.10.229.108:2117,34.10.229.108:2118,
...
```
This just a small snippet that has been truncated where all the IPs in the cluster are printed out.
Other things we might consider is adding the node a command is executing for to the log output to make it a bit easier to follow.
There might be other ways to make the log more readable that we should think about in terms of large clusters.
There are also no log levels, or verbosity levels in `roachprod`, so it's not easy to find errors, warning specifically. These are mostly hardcoded into the log output, sometimes with `WARN:` etc.
Jira issue: CRDB-52701
Contributor guide
Assessment
This issue has not been assessed yet.