cockroachdb / cockroachdb/cockroach
perf: consider reporting linux performance counters
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
On suitable hardware, linux' `perf_event_open` syscall allows retrieving low-level performance counters. These may be of interest for our benchmarking.
**Describe the solution you'd like**
Something like https://github.com/aclements/go-perfevent, which hooks up with Go unit tests. For example, we use this in `cockroachdb/swiss`:
https://github.com/cockroachdb/swiss/blob/2932b022f6df26dab1acf9a7c5f2b8187bd4c1cf/bench_test.go#L170
Performance counters will automatically be added to the benchmark results via `b.ReportMetric`.
_However_ this won't work easily for benchmarks which spawn multiple goroutines because `perfbench` doesn't collect counters for all threads (and goroutines multiplex over threads). Instead, it locks the current goroutine to an OS thread and collects counters for that thread only:
https://github.com/aclements/go-perfevent/blob/f34bb3e1a4e4dfc4f33f2ae468a82211eed7c2ab/perf/counter.go#L26-L42
It should be possible to extend `perfbench` with a `perf.Target` that attaches to all current and future threads managed by the Go runtime, see https://gist.github.com/tbg/b3504207374f92ccf0c2863dba97abf4.
----
As an (ugly) alternative: we could "externally" (`perf -e .... ./pkg/something.test -run ...`) collect these counters, but then we're collecting over the (usually higher-variance) setup phase as well. We could also invoke the `perf` tool externally, something like this:
```bash
#!/bin/bash
# Set the PID of the process to monitor
PID_TO_MONITOR=1234
# Create a temporary named pipe
FIFO=$(mktemp -u)
mkfifo "$FIFO"
# Start perf in the background
perf record -e instructions -p $PID_TO_MONITOR -o perf.data sh -c "cat $FIFO; kill -SIGINT \$\$" &
PERF_PID=$!
# Simulate the monitored process
(
# This is where your actual process (PID 1234) would run
# For demonstration, we'll sleep for 10 seconds then signal to stop
sleep 10
echo "stop" > "$FIFO"
)
# Wait for perf to finish
wait $PERF_PID
# Clean up
rm "$FIFO"
echo "Perf recording completed. Results in perf.data"
```
but it seems extremely messy.
Besides, on cloud VMs perf counters are often unavailable. on AWS for example, you get them on machine types which have a `.metal` analogue. These are usually expensive, beefy, VMs.
Jira issue: CRDB-49249
Contributor guide
Assessment
This issue has not been assessed yet.