Pi 3+ buster pxe nfsboot - kernel hangs / "Fatal exception in interrupt" panics

Start by reproducing PXE/NFS boot on a Raspberry Pi 3+ with the June 2019 Buster kernel and 4.19.57-v7+ #1244, then compare the complete boot and panic output. Focus on the ksoftirqd/0 trace, rcu_process_callbacks, and the listed NFS and networking boot stages. Done requires identifying the cause of the hang or panic and confirming a kernel fix under the same boot setup.

Describe the bug

PXE/nfs booting Buster on a Pi 3+ does not quite work

Does boot the kernel and starts to execute the first systemd units, but then the system hangs some seconds after on random moments (text cursor stops blinking, no activity, numlock does not respond)

Happens both with the kernel version you ship by default in your June 2019 Buster image, and 4.19.57 you get after running "apt-get upgrade"

In most cases kernels hangs without outputing any message, but seen the following kernel panic as well.

         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Mounted /tmp.
[  OK  ] Mounted /media.
[  OK  ] Mounted /var/lib/sudo.
[  OK  ] Mounted /var/lib/dhcp.
[  OK  ] Mounted /var/tmp.
[  OK  ] Mounted /home.
[  OK  ] Mounted /var/log.
[  OK  ] Mounted /var/cache.
[  OK  ] Mounted /var/spool.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Mounted /var/lib/systemd/rfkill.
[  OK  ] Mounted /var/lib/lightdm.
[  OK  ] Mounted /var/lib/systemd/coredump.
[  OK  ] Started Helper to synchronize boot up for ifupdown.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Reached target Local File Systems.
         Starting Preprocess NFS configuration...
         Starting Set console font and keymap...
         Starting Raise network interfaces...
[  OK  ] Started Preprocess NFS configuration.
[  OK  ] Reached target NFS client services.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
[   14.481793] Unable to handle kernel paging request at virtual address f74193c8
[   14.500886] pgd = 30af461a
[   14.515245] [f74193c8] *pgd=00000000
[   14.530421] Internal error: Oops: 80000005 [#1] SMP ARM
[   14.547260] Modules linked in: bcm2835_v4l2(C+) v4l2_mem2mem snd_timer v4l2_common bcm2835_mmal_vchiq(C) snd videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6
[   14.597249] CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G         C        4.19.57-v7+ #1244
[   14.617972] Hardware name: BCM2835
[   14.633734] PC is at 0xf74193c8
[   14.649171] LR is at rcu_process_callbacks+0x2cc/0x674
[   14.666604] pc : [<f74193c8>]    lr : [<8018e0b8>]    psr: a0000013
[   14.685285] sp : b9d3de58  ip : b9d3de58  fp : b9d3decc
[   14.702969] r10: b9d3c018  r9 : 0000000a  r8 : 80d10c80
[   14.720594] r7 : b9d3de8c  r6 : ba352660  r5 : 80d10780  r4 : ba352640
[   14.739565] r3 : f74193c8  r2 : f74193c8  r1 : b9d3de08  r0 : b7419654
[   14.758469] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   14.778024] Control: 10c5383d  Table: 376fc06a  DAC: 00000055
[   14.796173] Process ksoftirqd/0 (pid: 9, stack limit = 0x1631a8d3)
[   14.814825] Stack: (0xb9d3de58 to 0xb9d3e000)
[   14.831548] de40:                                                       00000000 80d04d48
[   14.852230] de60: ba351e40 00000001 80d10780 80d97784 80d04d48 80d04d70 80d8ed74 80c903f0
[   14.872765] de80: 00000001 80d04e10 80102374 f74193c8 b8350000 fffffffa ffffffff a271c001
[   14.893156] dea0: b9d3decc 80d030a4 00000009 b9d3c010 00000100 80d0be48 00000024 00000200
[   14.913429] dec0: b9d3df34 b9d3ded0 80102410 8018ddf8 802018e8 80202aec 00000000 04208040
[   14.933588] dee0: 80d03d00 ffff907a 00000009 80c94a00 80d8f278 80d04d70 80c8c308 80d03080
[   14.953722] df00: b9d3ded0 0000000a b9d3df34 b9d3c000 b9ccb380 00000000 00000001 80d0be48
[   14.973701] df20: 80d04d48 00000000 b9d3df44 b9d3df38 8012664c 8010228c b9d3df7c b9d3df48
[   14.993483] df40: 80146b1c 80126614 b9d40000 a271c001 b9d3df7c b9ccb580 b9ccb540 00000000
[   15.013156] df60: b9ccb380 801469e4 b9ccb59c b9d0ddd4 b9d3dfac b9d3df80 8014251c 801469f0
[   15.032816] df80: ffffffff b9ccb540 801423e4 00000000 00000000 00000000 00000000 00000000
[   15.052490] dfa0: 00000000 b9d3dfb0 801010ac 801423f0 00000000 00000000 00000000 00000000
[   15.072096] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   15.091679] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   15.111014] [<8018e0b8>] (rcu_process_callbacks) from [<80102410>] (__do_softirq+0x190/0x3f0)
[   15.130616] [<80102410>] (__do_softirq) from [<8012664c>] (run_ksoftirqd+0x44/0x54)
[   15.149185] [<8012664c>] (run_ksoftirqd) from [<80146b1c>] (smpboot_thread_fn+0x138/0x1d0)
[   15.168335] [<80146b1c>] (smpboot_thread_fn) from [<8014251c>] (kthread+0x138/0x168)
[   15.186934] [<8014251c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[   15.204883] Exception stack(0xb9d3dfb0 to 0xb9d3dff8)
[   15.220500] dfa0:                                     00000000 00000000 00000000 00000000
[   15.239418] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   15.258293] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   15.275462] Code: bad PC value
[   15.288934] ---[ end trace 3e6a1f362873c12f ]---
[   15.304036] Kernel panic - not syncing: Fatal exception in interrupt
[   15.321004] CPU1: stopping
[   15.334358] CPU: 1 PID: 144 Comm: systemd-udevd Tainted: G      D  C        4.19.57-v7+ #1244
[   15.353750] Hardware name: BCM2835
[   15.367836] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.386527] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.404422] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.422021] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.441271] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<80101d0c>] (__irq_usr+0x4c/0x60)
[   15.460543] Exception stack(0xb8269fb0 to 0xb8269ff8)
[   15.475907] 9fa0:                                     00000001 004f4910 2d4eb800 2d4eb800
[   15.494608] 9fc0: 0000002f 00000000 7ed3897c 76ffd968 004f4910 004e8680 0000002f 7ed38924
[   15.513331] 9fe0: 00515cbc 7ed38758 004c1998 76ee1134 60000010 ffffffff
[   15.530213] CPU3: stopping
[   15.542679] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D  C        4.19.57-v7+ #1244
[   15.560547] Hardware name: BCM2835
[   15.573622] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.591260] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.608438] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.625665] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.644624] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   15.663548] Exception stack(0xb9d51f38 to 0xb9d51f80)
[   15.678202] 1f20:                                                       80109a84 00000000
[   15.695751] 1f40: 40000193 40000193 80d04d70 b9d50000 80d04db8 00000008 80d8ed3e 410fd034
[   15.713290] 1f60: 00000000 b9d51f94 80d0517c b9d51f88 00000000 80109a88 40000113 ffffffff
[   15.730839] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   15.747679] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   15.765310] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   15.782510] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   15.799636] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   15.818384] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   15.835311] CPU2: stopping
[   15.847637] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D  C        4.19.57-v7+ #1244
[   15.865445] Hardware name: BCM2835
[   15.878460] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.896045] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.913200] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.930420] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.949258] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   15.967977] Exception stack(0xb9d4ff38 to 0xb9d4ff80)
[   15.982797] ff20:                                                       80109a84 00000000
[   16.000859] ff40: 40000093 40000093 80d04d70 b9d4e000 80d04db8 00000004 80d8ed3e 410fd034
[   16.018940] ff60: 00000000 b9d4ff94 80d0517c b9d4ff88 00000000 80109a88 40000013 ffffffff
[   16.037026] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   16.054360] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   16.072432] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   16.090080] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   16.107627] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   16.126830] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   16.144261] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---