adoptium / adoptium/infrastructure

High load on `test-osuosl-ubuntu1604-ppc64le-2`

Open
#2,776 4 comments 0 reactions 0 assignees View on GitHub
arch:ppc64le (pLinux) provider:osuosl
Dominant language
Python
Stars
96
Forks
106
Avg merge
1d 23h
Merged PRs (30d)
13

Description

We are getting warning messages from Nagios that the machine is sitting with a load of 17.00:
`HOST: test-osuosl-ubuntu1604-ppc64le-2 SERVICE: Current Load STATE: WARNING MESSAGE: WARNING - load average: 17.00, 17.00, 17.00 [See Nagios](https://nagios.adoptopenjdk.net/nagios/cgi-bin/status.cgi?host=test-osuosl-ubuntu1604-ppc64le-2)`

Note that while it's currently stuck up there at 0904 this morning Nagios declared it good again with load averages of `0.04 0.05 1.02`, but then it went up again.

This machine running Ubuntu 16.04.7 and has been online for over a year:
```
10:48:32 up 337 days, 16:52, 1 user, load average: 17.14, 17.07, 17.02
```
There are no obvious processes using lots of CPU, although there has been a recent kernel exception:
```
[29108237.051171] kernel BUG at /build/linux-6rygVt/linux-4.4.0/mm/memory.c:3214!
[29108237.051730] Oops: Exception in kernel mode, sig: 5 [#1]
[29108237.051799] SMP NR_CPUS=2048 NUMA pSeries
[29108237.051889] Modules linked in: ufs msdos xfs ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay aufs input_leds joydev vmx_crypto gf128mul ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core binfmt_misc ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid
[29108237.053506] CPU: 2 PID: 3090 Comm: java Not tainted 4.4.0-210-generic #242-Ubuntu
[29108237.053611] task: c0000001fa94ff00 ti: c000000019d28000 task.ti: c000000019d28000
[29108237.053714] NIP: c0000000002803f4 LR: c00000000027fb64 CTR: 0000000000000000
[29108237.053816] REGS: c000000019d2b440 TRAP: 0700 Not tainted (4.4.0-210-generic)
[29108237.053918] MSR: 8000000002029033 CR: 44822882 XER: 00000000
[29108237.054179] CFAR: c00000000027ffa8 SOFTE: 1
GPR00: c00000000027fb30 c000000019d2b6c0 c000000001654800 0000000000000001
GPR04: c000000003bf7ff8 00003fff72a30000 c00000018d8ae000 0000000000000000
GPR08: 0000000000000000 0000000000000001 bfffffffffffffff 0000000000000060
GPR12: 0000000024822882 c00000000fb01400 0000000000000000 c00000000156b7b3
GPR16: fffffffffffff000 00000000000000fd 0000000080000000 c00000018d8ae518
GPR20: c00000017c801b90 0000000000000000 c00000018d8ae000 0000000000000518
GPR24: 000050fa40000181 0000000000000001 0000000000001b90 c0000001fa9e2100
GPR28: 0000000000000000 c00000017c950578 00003fff72a30000 c00000017c800000
[29108237.055609] NIP [c0000000002803f4] handle_mm_fault+0x974/0x1940
[29108237.055697] LR [c00000000027fb64] handle_mm_fault+0xe4/0x1940
[29108237.055782] Call Trace:
[29108237.055819] [c000000019d2b6c0] [c00000000027fb30] handle_mm_fault+0xb0/0x1940 (unreliable)
[29108237.055980] [c000000019d2b790] [c000000000278b50] __get_user_pages+0x1a0/0x550
[29108237.056102] [c000000019d2b840] [c0000000002798fc] get_dump_page+0x4c/0x80
[29108237.056204] [c000000019d2b880] [c000000000375dd0] elf_core_dump+0x800/0x8e0
[29108237.056327] [c000000019d2ba60] [c00000000037e90c] do_coredump+0xddc/0x1250
[29108237.056435] [c000000019d2bc20] [c0000000000d4a70] get_signal+0x1b0/0x9e0
[29108237.056539] [c000000019d2bd10] [c00000000001a988] do_signal+0x68/0x2c0
[29108237.056668] [c000000019d2be00] [c00000000001addc] do_notify_resume+0xbc/0xd0
[29108237.056789] [c000000019d2be30] [c00000000000bf38] ret_from_except_lite+0x64/0x68
[29108237.056909] Instruction dump:
[29108237.056979] 912a0004 7ea3ab78 4bfce77d 60000000 4bfff88c 60000000 60420000 e93d0050
[29108237.057158] 571c05ac 79290760 7d290074 7929d182 <0b090000> 7c7af82a 4bdd930d 60000000
[29108240.947198] ---[ end trace 82a11c1081695ed3 ]---

root@test-osuosl-ubuntu1604-ppc64le-2:~# cat /proc/uptime
29177874.98 116091099.58
root@test-osuosl-ubuntu1604-ppc64le-2:~# uptime
10:54:42 up 337 days, 16:59, 1 user, load average: 18.38, 17.93, 17.40
root@test-osuosl-ubuntu1604-ppc64le-2:~#

```
I've kicked off https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk8_hs_sanity.openjdk_ppc64le_linux/742/ to see if the machine is actually acting slow due to the high load, but I expect a reboot will be in order.

Contributor guide

Open the contributing guide

Research direction

Start with the Nagios Current Load alert for test-osuosl-ubuntu1604-ppc64le-2, the recorded kernel BUG in handle_mm_fault, and the linked Test_openjdk8_hs_sanity.openjdk_ppc64le_linux build. Check whether the host remains responsive and whether the load is explained by the kernel exception; done means identifying the cause and documenting or carrying out the required recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux, ubuntu
Domain
infrastructure, observability-sre, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.