cockroachdb / cockroachdb/cockroach

perf: consider reporting linux performance counters

Open
#144,137 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-perf C-enhancement o-perf-efficiency P-3 T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

On suitable hardware, linux' `perf_event_open` syscall allows retrieving low-level performance counters. These may be of interest for our benchmarking.

**Describe the solution you'd like**

Something like https://github.com/aclements/go-perfevent, which hooks up with Go unit tests. For example, we use this in `cockroachdb/swiss`:

https://github.com/cockroachdb/swiss/blob/2932b022f6df26dab1acf9a7c5f2b8187bd4c1cf/bench_test.go#L170

Performance counters will automatically be added to the benchmark results via `b.ReportMetric`.

_However_ this won't work easily for benchmarks which spawn multiple goroutines because `perfbench` doesn't collect counters for all threads (and goroutines multiplex over threads). Instead, it locks the current goroutine to an OS thread and collects counters for that thread only:

https://github.com/aclements/go-perfevent/blob/f34bb3e1a4e4dfc4f33f2ae468a82211eed7c2ab/perf/counter.go#L26-L42

It should be possible to extend `perfbench` with a `perf.Target` that attaches to all current and future threads managed by the Go runtime, see https://gist.github.com/tbg/b3504207374f92ccf0c2863dba97abf4.

----

As an (ugly) alternative: we could "externally" (`perf -e .... ./pkg/something.test -run ...`) collect these counters, but then we're collecting over the (usually higher-variance) setup phase as well. We could also invoke the `perf` tool externally, something like this:

```bash
#!/bin/bash

# Set the PID of the process to monitor
PID_TO_MONITOR=1234

# Create a temporary named pipe
FIFO=$(mktemp -u)
mkfifo "$FIFO"

# Start perf in the background
perf record -e instructions -p $PID_TO_MONITOR -o perf.data sh -c "cat $FIFO; kill -SIGINT \$\$" &
PERF_PID=$!

# Simulate the monitored process
(
# This is where your actual process (PID 1234) would run
# For demonstration, we'll sleep for 10 seconds then signal to stop
sleep 10
echo "stop" > "$FIFO"
)

# Wait for perf to finish
wait $PERF_PID

# Clean up
rm "$FIFO"

echo "Perf recording completed. Results in perf.data"

```

but it seems extremely messy.

Besides, on cloud VMs perf counters are often unavailable. on AWS for example, you get them on machine types which have a `.metal` analogue. These are usually expensive, beefy, VMs.

Jira issue: CRDB-49249

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.