Linux guests unable to capture hardware events using `perf record`

Linux guests on colo (R13) and dogfood (R14rc0) are unable to collect hardware events using perf record.

The standard events pulled by perf record will return EOPNOTSUPP on perf_event_open. Selecting specific hardware events, e.g. cpu-cycles, instructions, but still don't get any samples. Software events work normally, as does perf stat.

I've reproduced this on Debian 12 and Ubuntu 24.04 guests.

Without specifying events to record, perf_event_open returns EOPNOTSUPP

ubuntu@perf:~$ uname -r
6.8.0-31-generic

ubuntu@perf:~$ sudo perf record -vvvv ls
DEBUGINFOD_URLS=
Using CPUID AuthenticAMD-25-1-1
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 0
perf record opening and mmapping events
Opening: cycles:P
------------------------------------------------------------
perf_event_attr:
  type                             0 (PERF_TYPE_HARDWARE)
  size                             136
  config                           0 (PERF_COUNT_HW_CPU_CYCLES)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|PERIOD
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  mmap                             1
  comm                             1
  freq                             1
  enable_on_exec                   1
  task                             1
  precise_ip                       3
  sample_id_all                    1
  mmap2                            1
  comm_exec                        1
  ksymbol                          1
  bpf_event                        1
------------------------------------------------------------
sys_perf_event_open: pid 1685  cpu 0  group_fd -1  flags 0x8
sys_perf_event_open failed, error -95
...

ubuntu@perf:~$ sudo perf report --stdio
Error:
The perf.data data has no samples!

Specific hardware event

ubuntu@perf:~$ sudo perf record -e cpu-cycles -vvvv -a sleep 1
DEBUGINFOD_URLS=
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 0
perf record opening and mmapping events
Opening: cpu-cycles
------------------------------------------------------------
perf_event_attr:
  type                             0 (PERF_TYPE_HARDWARE)
  size                             136
  config                           0 (PERF_COUNT_HW_CPU_CYCLES)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|CPU|PERIOD|IDENTIFIER
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  freq                             1
  sample_id_all                    1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8 = 5
sys_perf_event_open: pid -1  cpu 1  group_fd -1  flags 0x8 = 6
...
ubuntu@perf:~$ sudo perf report --stdio
Error:
The perf.data data has no samples!

Software events works normally

ubuntu@perf:~$ sudo perf record -e cpu-clock -vvvv -a sleep 1
DEBUGINFOD_URLS=
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 0
perf record opening and mmapping events
Opening: cpu-clock
------------------------------------------------------------
perf_event_attr:
  type                             1 (PERF_TYPE_SOFTWARE)
  size                             136
  config                           0 (PERF_COUNT_SW_CPU_CLOCK)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|CPU|PERIOD|IDENTIFIER
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  freq                             1
  sample_id_all                    1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8 = 5
sys_perf_event_open: pid -1  cpu 1  group_fd -1  flags 0x8 = 6
...
ubuntu@perf:~$ sudo perf report --stdio
# Samples: 2K of event 'cpu-clock'
# Event count (approx.): 668250000
...

perf stat also works:

ubuntu@perf:~$ sudo perf stat sleep 1

 Performance counter stats for 'sleep 1':

              3.12 msec task-clock                       #    0.003 CPUs utilized             
                 2      context-switches                 #  641.491 /sec                      
                 0      cpu-migrations                   #    0.000 /sec                      
                78      page-faults                      #   25.018 K/sec                     
           1885037      cycles                           #    0.605 GHz                         (12.71%)
                 0      stalled-cycles-frontend                                                 (40.24%)
                 0      stalled-cycles-backend                                                  (66.53%)
           2021096      instructions                     #    1.07  insn per cycle              (87.29%)
            432471      branches                         #  138.713 M/sec                       (59.76%)
             45945      branch-misses                    #   10.62% of all branches             (33.47%)

       1.004869669 seconds time elapsed

       0.000000000 seconds user
       0.004696000 seconds sys