adoptium / adoptium/infrastructure
High load on `test-osuosl-ubuntu1604-ppc64le-2`
- Dominant language
- Python
- Stars
- 96
- Forks
- 106
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
We are getting warning messages from Nagios that the machine is sitting with a load of 17.00:
`HOST: test-osuosl-ubuntu1604-ppc64le-2 SERVICE: Current Load STATE: WARNING MESSAGE: WARNING - load average: 17.00, 17.00, 17.00 [See Nagios](https://nagios.adoptopenjdk.net/nagios/cgi-bin/status.cgi?host=test-osuosl-ubuntu1604-ppc64le-2)`
Note that while it's currently stuck up there at 0904 this morning Nagios declared it good again with load averages of `0.04 0.05 1.02`, but then it went up again.
This machine running Ubuntu 16.04.7 and has been online for over a year:
```
10:48:32 up 337 days, 16:52, 1 user, load average: 17.14, 17.07, 17.02
```
There are no obvious processes using lots of CPU, although there has been a recent kernel exception:
```
[29108237.051171] kernel BUG at /build/linux-6rygVt/linux-4.4.0/mm/memory.c:3214!
[29108237.051730] Oops: Exception in kernel mode, sig: 5 [#1]
[29108237.051799] SMP NR_CPUS=2048 NUMA pSeries
[29108237.051889] Modules linked in: ufs msdos xfs ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay aufs input_leds joydev vmx_crypto gf128mul ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core binfmt_misc ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid
[29108237.053506] CPU: 2 PID: 3090 Comm: java Not tainted 4.4.0-210-generic #242-Ubuntu
[29108237.053611] task: c0000001fa94ff00 ti: c000000019d28000 task.ti: c000000019d28000
[29108237.053714] NIP: c0000000002803f4 LR: c00000000027fb64 CTR: 0000000000000000
[29108237.053816] REGS: c000000019d2b440 TRAP: 0700 Not tainted (4.4.0-210-generic)
[29108237.053918] MSR: 8000000002029033 CR: 44822882 XER: 00000000
[29108237.054179] CFAR: c00000000027ffa8 SOFTE: 1
GPR00: c00000000027fb30 c000000019d2b6c0 c000000001654800 0000000000000001
GPR04: c000000003bf7ff8 00003fff72a30000 c00000018d8ae000 0000000000000000
GPR08: 0000000000000000 0000000000000001 bfffffffffffffff 0000000000000060
GPR12: 0000000024822882 c00000000fb01400 0000000000000000 c00000000156b7b3
GPR16: fffffffffffff000 00000000000000fd 0000000080000000 c00000018d8ae518
GPR20: c00000017c801b90 0000000000000000 c00000018d8ae000 0000000000000518
GPR24: 000050fa40000181 0000000000000001 0000000000001b90 c0000001fa9e2100
GPR28: 0000000000000000 c00000017c950578 00003fff72a30000 c00000017c800000
[29108237.055609] NIP [c0000000002803f4] handle_mm_fault+0x974/0x1940
[29108237.055697] LR [c00000000027fb64] handle_mm_fault+0xe4/0x1940
[29108237.055782] Call Trace:
[29108237.055819] [c000000019d2b6c0] [c00000000027fb30] handle_mm_fault+0xb0/0x1940 (unreliable)
[29108237.055980] [c000000019d2b790] [c000000000278b50] __get_user_pages+0x1a0/0x550
[29108237.056102] [c000000019d2b840] [c0000000002798fc] get_dump_page+0x4c/0x80
[29108237.056204] [c000000019d2b880] [c000000000375dd0] elf_core_dump+0x800/0x8e0
[29108237.056327] [c000000019d2ba60] [c00000000037e90c] do_coredump+0xddc/0x1250
[29108237.056435] [c000000019d2bc20] [c0000000000d4a70] get_signal+0x1b0/0x9e0
[29108237.056539] [c000000019d2bd10] [c00000000001a988] do_signal+0x68/0x2c0
[29108237.056668] [c000000019d2be00] [c00000000001addc] do_notify_resume+0xbc/0xd0
[29108237.056789] [c000000019d2be30] [c00000000000bf38] ret_from_except_lite+0x64/0x68
[29108237.056909] Instruction dump:
[29108237.056979] 912a0004 7ea3ab78 4bfce77d 60000000 4bfff88c 60000000 60420000 e93d0050
[29108237.057158] 571c05ac 79290760 7d290074 7929d182 <0b090000> 7c7af82a 4bdd930d 60000000
[29108240.947198] ---[ end trace 82a11c1081695ed3 ]---
root@test-osuosl-ubuntu1604-ppc64le-2:~# cat /proc/uptime
29177874.98 116091099.58
root@test-osuosl-ubuntu1604-ppc64le-2:~# uptime
10:54:42 up 337 days, 16:59, 1 user, load average: 18.38, 17.93, 17.40
root@test-osuosl-ubuntu1604-ppc64le-2:~#
```
I've kicked off https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk8_hs_sanity.openjdk_ppc64le_linux/742/ to see if the machine is actually acting slow due to the high load, but I expect a reboot will be in order.
Contributor guide
Research direction
Start with the Nagios Current Load alert for test-osuosl-ubuntu1604-ppc64le-2, the recorded kernel BUG in handle_mm_fault, and the linked Test_openjdk8_hs_sanity.openjdk_ppc64le_linux build. Check whether the host remains responsive and whether the load is explained by the kernel exception; done means identifying the cause and documenting or carrying out the required recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux, ubuntu
- Domain
- infrastructure, observability-sre, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100