On s390x, test_kernel_stat and test_meminfo fail

I originally saw this while working on an update for the [`rust-procfs` package in Fedora](https://src.fedoraproject.org/rpms/rust-procfs); the package build still happens in a `mock` chroot, but on real `s390x` hardware rather than under `qemu-user-static` emulation. I’ve tried to reproduce it in emulation, below, since I suspect that may be more useful to you.

----

Working in a Fedora 41 [`mock` chroot](https://github.com/rpm-software-management/mock) with `git`, `cargo`, and `rust` installed, and with network access enabled:

On `x86_64`, skipping:

- `process::tests::test_proc_fd_count_runsinglethread` because of https://github.com/eminence/procfs/issues/322
- `sys::kernel::random::tests::test_write_wakeup_threshold`, `process::tests::test_proc_auxv`, and `process::tests::test_proc_status_for_kthreadd` because they fail in a mock chroot for various reasons

```
$ uname -srvmo
Linux 6.11.7-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 8 19:23:10 UTC 2024 x86_64 GNU/Linux
$ git clone https://github.com/eminence/procfs.git
$ cd procfs
$ git checkout v0.17.0
$ RUST_BACKTRACE=1 cargo test -- --skip-exact \
--skip=process::tests::test_proc_fd_count_runsinglethread \
--skip sys::kernel::random::tests::test_write_wakeup_threshold \
--skip process::tests::test_proc_auxv \
--skip=process::tests::test_proc_status_for_kthreadd
(all tests pass or are skipped/ignored)
```

This is our “control” build.

Now, repeating in an `aarch64` chroot, emulated via `qemu-user-static`:

```
$ RUST_BACKTRACE=1 cargo test -- --skip-exact […]
[…]
failures:

---- process::tests::test_proc_exe stdout ----
thread 'process::tests::test_proc_exe' panicked at procfs/src/process/tests.rs:275:5:
assertion `left == right` failed
left: "/usr/bin/qemu-aarch64-static"
right: "/procfs/target/debug/deps/procfs-daeabd3c8934810d"
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
4: procfs::process::tests::test_proc_exe
at ./src/process/tests.rs:275:5
5: procfs::process::tests::test_proc_exe::{{closure}}
at ./src/process/tests.rs:271:19
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at procfs/src/lib.rs:684:33:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:148:5
3: core::option::unwrap_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:2012:5
4: core::option::Option::unwrap
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:972:21
5: procfs::tests::test_cpuinfo
at ./src/lib.rs:684:13
6: procfs::tests::test_cpuinfo::{{closure}}
at ./src/lib.rs:679:22
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
8: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- process::tests::test_procinfo stdout ----
thread 'process::tests::test_procinfo' panicked at procfs/src/process/tests.rs:506:52:
called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: "unable to parse input: Position(Space, [10])" }
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::result::unwrap_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1700:5
3: core::result::Result::unwrap
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1104:23
4: procfs::process::tests::test_procinfo
at ./src/process/tests.rs:506:25
5: procfs::process::tests::test_procinfo::{{closure}}
at ./src/process/tests.rs:494:19
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

failures:
process::tests::test_proc_exe
process::tests::test_procinfo
tests::test_cpuinfo

test result: FAILED. 91 passed; 3 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.27s
```

Okay, the `tests::test_cpuinfo` failure is already reported as https://github.com/eminence/procfs/issues/323, the `process::tests::test_proc_exe` failure seems to be specific to running in emulation, and I don’t know what to say about `process::tests::test_procinfo`. I didn’t see it in a “real” Fedora build, and none of these are what I am trying to report here.

Repeating the above on `ppc64le` gives the same result, except that `tests::test_cpuinfo` doesn’t fail – although it *did* fail on real `ppc64le` hardware.

----

Now, for the subject of this report, on `s390x`:

```
[…]
---- tests::test_kernel_stat stdout ----
KernelStats {
total: CpuTime {
user: 83030541,
nice: 1456170471,
system: 13524974,
idle: 3154050570,
iowait: Some(
1543050,
),
irq: Some(
9041546,
),
softirq: Some(
866113,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
cpu_time: [
CpuTime {
user: 5197507,
nice: 90828479,
system: 836449,
idle: 197307712,
iowait: Some(
74260,
),
irq: Some(
558475,
),
softirq: Some(
82892,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4596389,
nice: 91197844,
system: 674628,
idle: 197775509,
iowait: Some(
62124,
),
irq: Some(
548304,
),
softirq: Some(
44540,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3530071,
nice: 90147038,
system: 742351,
idle: 199879912,
iowait: Some(
41213,
),
irq: Some(
526190,
),
softirq: Some(
42161,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 8423397,
nice: 91541602,
system: 1349167,
idle: 192570622,
iowait: Some(
268915,
),
irq: Some(
620041,
),
softirq: Some(
54695,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4052862,
nice: 90933503,
system: 615937,
idle: 198685170,
iowait: Some(
50074,
),
irq: Some(
536112,
),
softirq: Some(
38925,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3940753,
nice: 90860186,
system: 586909,
idle: 198930525,
iowait: Some(
26126,
),
irq: Some(
537879,
),
softirq: Some(
36786,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4180428,
nice: 91013747,
system: 615564,
idle: 198459494,
iowait: Some(
45197,
),
irq: Some(
556439,
),
softirq: Some(
38756,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 8331391,
nice: 91488962,
system: 1236338,
idle: 192896853,
iowait: Some(
224505,
),
irq: Some(
612317,
),
softirq: Some(
53209,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5169403,
nice: 90916745,
system: 843821,
idle: 197260698,
iowait: Some(
75680,
),
irq: Some(
564525,
),
softirq: Some(
49986,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 10122500,
nice: 91362720,
system: 1804961,
idle: 190432249,
iowait: Some(
359176,
),
irq: Some(
652213,
),
softirq: Some(
95041,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4263470,
nice: 91060333,
system: 718679,
idle: 198125533,
iowait: Some(
74775,
),
irq: Some(
565820,
),
softirq: Some(
87917,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5419741,
nice: 91275798,
system: 862684,
idle: 196556631,
iowait: Some(
95307,
),
irq: Some(
590100,
),
softirq: Some(
67471,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3962060,
nice: 90859637,
system: 655854,
idle: 198777343,
iowait: Some(
44810,
),
irq: Some(
547991,
),
softirq: Some(
62077,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3457406,
nice: 90381010,
system: 744833,
idle: 199720763,
iowait: Some(
38593,
),
irq: Some(
530356,
),
softirq: Some(
37793,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4528047,
nice: 91232485,
system: 645171,
idle: 197878344,
iowait: Some(
32209,
),
irq: Some(
554304,
),
softirq: Some(
37576,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3855107,
nice: 91070372,
system: 591621,
idle: 198793205,
iowait: Some(
30079,
),
irq: Some(
540472,
),
softirq: Some(
36280,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
],
ctxt: 3912665812,
btime: 1731761843,
processes: 7953721,
procs_running: Some(
4,
),
procs_blocked: Some(
0,
),
}
thread 'tests::test_kernel_stat' panicked at procfs/src/lib.rs:577:9:
assertion `left == right` failed
left: 1
right: 16
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
4: procfs::tests::test_kernel_stat
at ./src/lib.rs:577:9
5: procfs::tests::test_kernel_stat::{{closure}}
at ./src/lib.rs:566:26
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[…]

failures:
process::tests::test_proc_exe
process::tests::test_procinfo
tests::test_cpuinfo
tests::test_kernel_stat

test result: FAILED. 90 passed; 4 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.18s
```

The failures in `process::tests::test_proc_exe`, `process::tests::test_procinfo`, and `tests::test_cpuinfo` are consistent with those on `aarch64` and `pp664le`; `tests::test_kernel_stat` is one of the failures I was trying to report in this issue. The output on real hardware is similar:

```
---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at src/lib.rs:683:34:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- tests::test_kernel_stat stdout ----
KernelStats {
total: CpuTime {
user: 16790930,
nice: 3522,
system: 3384651,
idle: 233015124,
iowait: Some(
326403,
),
irq: Some(
107465,
),
softirq: Some(
185327,
),
steal: Some(
173507,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
cpu_time: [
CpuTime {
user: 5522045,
nice: 917,
system: 1138502,
idle: 77678704,
iowait: Some(
103183,
),
irq: Some(
53849,
),
softirq: Some(
111186,
),
steal: Some(
58703,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5627933,
nice: 1041,
system: 1124884,
idle: 77670809,
iowait: Some(
113340,
),
irq: Some(
27253,
),
softirq: Some(
37545,
),
steal: Some(
57362,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5640952,
nice: 1563,
system: 1121263,
idle: 77665610,
iowait: Some(
109879,
),
irq: Some(
26363,
),
softirq: Some(
36595,
),
steal: Some(
57441,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
],
ctxt: 555939405,
btime: 1733854463,
processes: 13904974,
procs_running: Some(
3,
),
procs_blocked: Some(
0,
),
}
thread 'tests::test_kernel_stat' panicked at src/lib.rs:577:9:
assertion `left == right` failed
left: 1
right: 3
```

Unfortunately, I wasn’t able to reproduce the `test_meminfo` failure in emulation, but it looks like this on real hardware:

```
---- tests::test_meminfo stdout ----
Meminfo {
mem_total: 26335961088,
mem_free: 4957818880,
mem_available: Some(
25220423680,
),
buffers: 1196032,
cached: 19969449984,
swap_cached: 16384,
active: 5222580224,
inactive: 14961217536,
active_anon: Some(
169172992,
),
inactive_anon: Some(
44261376,
),
active_file: Some(
5053407232,
),
inactive_file: Some(
14916956160,
),
unevictable: Some(
0,
),
mlocked: Some(
0,
),
high_total: None,
high_free: None,
low_total: None,
low_free: None,
mmap_copy: None,
swap_total: 8589930496,
swap_free: 8589144064,
dirty: 666693632,
writeback: 0,
anon_pages: Some(
213184512,
),
mapped: 278470656,
shmem: Some(
278528,
),
slab: 830554112,
s_reclaimable: Some(
554332160,
),
s_unreclaim: Some(
276221952,
),
kernel_stack: Some(
15982592,
),
page_tables: Some(
44687360,
),
secondary_page_tables: Some(
0,
),
quicklists: None,
nfs_unstable: Some(
0,
),
bounce: Some(
0,
),
writeback_tmp: Some(
0,
),
commit_limit: Some(
21757911040,
),
committed_as: 579280896,
vmalloc_total: 547608330240,
vmalloc_used: 64446464,
vmalloc_chunk: 0,
hardware_corrupted: None,
anon_hugepages: None,
shmem_hugepages: None,
shmem_pmd_mapped: None,
cma_total: Some(
0,
),
cma_free: Some(
0,
),
hugepages_total: Some(
0,
),
hugepages_free: Some(
0,
),
hugepages_rsvd: Some(
0,
),
hugepages_surp: Some(
0,
),
hugepagesize: Some(
1048576,
),
direct_map_4k: Some(
7832862720,
),
direct_map_4M: None,
direct_map_2M: None,
direct_map_1G: None,
hugetlb: Some(
0,
),
per_cpu: Some(
20029440,
),
k_reclaimable: Some(
554332160,
),
file_pmd_mapped: None,
file_huge_pages: None,
z_swap: Some(
0,
),
z_swapped: Some(
0,
),
}
thread 'tests::test_meminfo' panicked at src/lib.rs:848:13:
assertion failed: meminfo.hardware_corrupted.is_some()
```

Reproduce the report on real s390x hardware using the Fedora 41 mock-chroot setup and the cargo test command described in the issue. Start with tests::test_kernel_stat and tests::test_meminfo, then compare their failures with the reported s390x output and the existing procfs parsing behavior. Done means the relevant tests pass on s390x without regressing the x86_64 control run.

Written by the indexing model from the issue text.