Unable to handle kernel NULL pointer dereference at virtual address 00000028

Start at the bcm2835_sdhost_dma_complete entry point shown in the trace and compare the reported 4.19.118-v7+ and 5.15.21-v7+ kernel behavior on the Raspberry Pi 2B. Reproduce the update and reboot sequence using rpi-update or apt full-upgrade, then determine what change prevents the NULL-pointer panic before login.

Describe the bug

Post updating a system running 2020-05-27-raspios-buster-armhf using kernel7.img (this is the only image archive that I can get to boot without this crash), the following system crash occurs on reboot. The crash is the same regardless of update method:

• sudo rpi-update
• Using sudo apt full-upgrade -y

Steps to reproduce the behaviour

1. Boot system with 2020-05-27-raspios-buster-armhf and kernel7.img on a RPI 2B
2. Update the system via sudo rpi-update or sudo apt full-upgrade -y

Device (s)

Raspberry Pi 2 Mod. B

System

System crashes before login post upgrade. This would only be valid for the system state before the upgrade.

Logs

Before Upgrade:

Linux raspberrypi 4.19.118-v7+ #1311 SMP Mon Apr 27 14:21:24 BST 2020 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Feb  7 22:22:38 2022
pi@raspberrypi:~ $ sudo rpi-update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
 *** Backing up modules 4.19.118-v7+
#############################################################
WARNING: This update bumps to rpi-5.15.y linux tree
See: https://forums.raspberrypi.com/viewtopic.php?t=322879

'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.
##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   173  100   173    0     0    239      0 --:--:-- --:--:-- --:--:--   252
100  123M    0  123M    0     0   143k      0 --:--:--  0:14:38 --:--:--  179k
 *** Updating firmware
 *** Updating kernel modules
 *** depmod 5.15.21-v8+
 *** depmod 5.15.21-v7l+
 *** depmod 5.15.21+
 *** depmod 5.15.21-v7+
 *** Updating VideoCore libraries
 *** Using HardFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to fbcab73d5a20f592aa9f6b0e364757ef131dae27
 *** A reboot is needed to activate the new firmware
pi@raspberrypi:~ $ sudo reboot

Console Log post reboot:

Raspbian GNU/Linux 10 raspberrypi ttyAMA0

raspberrypi login: usbnet: failed control transaction: request 0x2143 value 0xe index 0x0 length 0x0
usbnet: failed control transaction: request 0x2143 value 0xe index 0x0 length 0x0
usbnet: failed control transaction: request 0x2143 value 0xe index 0x0 length 0x0
usbnet: failed control transaction: request 0x2143 value 0xe index 0x0 length 0x0
usbnet: failed control transaction: request 0x2143 value 0xe index 0x0 length 0x0
[  818.554169] 8<--- cut here ---
[  818.555048] Unable to handle kernel NULL pointer dereference at virtual address 00000028
[  818.556593] pgd = c71a5b8e
[  818.557910] [00000028] *pgd=05e99835, *pte=00000000, *ppte=00000000
[  818.560379] Internal error: Oops: 17 [#1] SMP ARM
[  818.561895] Modules linked in: sha256_generic cfg80211 rfkill 8021q garp stp llc cdc_subset cdc_ether raspberrypi_hwmon fixed uio_pdrv_genirq uio i2c_dev ip_tables x_tables ipv6
[  818.588463] CPU: 0 PID: 451 Comm: dhcpcd Tainted: G        W         5.15.21-v7+ #1523
[  818.601158] Hardware name: BCM2835
[  818.610290] PC is at bcm2835_sdhost_dma_complete+0x9c/0x1f4
[  818.616447] LR is at log_event_impl.part.0+0x90/0x94
[  818.624539] pc : [<80858b28>]    lr : [<80857200>]    psr: a0000193
[  818.625626] sp : 84dcbde8  ip : 84dcbdc8  fp : 84dcbe14
[  818.626515] r10: ffffe000  r9 : 00000100  r8 : 00000000
[  818.627389] r7 : 00000000  r6 : a0000113  r5 : 810b7be4  r4 : 81fe0340
[  818.630426] r3 : 81665640  r2 : 00000000  r1 : a0000193  r0 : 810b7be4
[  818.636684] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  818.644425] Control: 10c5387d  Table: 0350006a  DAC: 00000055
[  818.651220] Register r0 information: non-slab/vmalloc memory
[  818.663061] Register r1 information: non-slab/vmalloc memory
[  818.669901] Register r2 information: NULL pointer
[  818.677481] Register r3 information: slab kmalloc-512 start 81665600 pointer offset 64 size 512
[  818.694985] Register r4 information: slab kmalloc-2k start 81fe0000 pointer offset 832 size 2048
[  818.705935] Register r5 information: non-slab/vmalloc memory
[  818.715669] Register r6 information: non-slab/vmalloc memory
[  818.724035] Register r7 information: NULL pointer
[  818.730534] Register r8 information: NULL pointer
[  818.741500] Register r9 information: non-paged memory
[  818.748417] Register r10 information: non-paged memory
[  818.755525] Register r11 information: non-slab/vmalloc memory
[  818.764784] Register r12 information: non-slab/vmalloc memory
[  818.776033] Process dhcpcd (pid: 451, stack limit = 0x03f2fd31)
[  818.785808] Stack: (0x84dcbde8 to 0x84dcc000)
[  818.794660] bde0:                   8022e900 8022fcdc 81665480 836d0884 836d08bc 84dcbde8
[  818.804773] be00: 84dcbe20 00000122 84dcbe6c 84dcbe18 806e0084 80858a98 80f05008 84dcbea4
[  818.814332] be20: 84dcbe20 84dcbe20 00000000 00000000 80858a8c 00000000 81fe0340 5674124e
[  818.823567] be40: 84dcbe6c 81665480 81fe0364 b77142d4 00000000 00000006 84dca000 ffffe000
[  818.831328] be60: 84dcbe94 84dcbe70 80126298 806dfeb0 80f03098 00000005 00000006 00000100
[  818.838040] be80: 000000b0 84dca000 84dcbea4 84dcbe98 801262dc 801261c4 84dcbf14 84dcbea8
[  818.848476] bea0: 80101468 801262c0 00000080 5674124e b771564c 80b0211c 00400040 00000001
[  818.857851] bec0: 80d0c4f0 80f03d00 0000c9de 80e8a2cc 0000000a 80e91b00 81038494 80f05058
[  818.870055] bee0: 80e8a348 80f03080 8018db08 ffffe000 00000000 00000000 00000008 60000010
[  818.880996] bf00: 84dca000 10c5387d 84dcbf2c 84dcbf18 80126088 801012dc 80e9110c 00000000
[  818.887252] bf20: 84dcbf4c 84dcbf30 8018822c 80125fbc 80100124 60000013 ffffffff 84dcbf94
[  818.899077] bf40: 84dcbf5c 84dcbf50 80101228 801881c8 00000000 84dcbf60 80100aac 801011d8
[  818.905855] bf60: 0000001e 76fa9d7c 3688a000 00000000 00000000 76fa9f48 0005dca0 000000ae
[  818.919999] bf80: 60000010 76e1e02c 10c5387d 00000000 84dcbfb0 84dcbfb0 80100120 80100124
[  818.934840] bfa0: 60000013 ffffffff 80100120 8022e8b4 0000001e 76fa9d7c 00000000 00000008
[  818.944293] bfc0: 00000000 76fa9f48 0005dca0 000000ae 0000001e 0000001e 7ede98b4 76fa9ffc
[  818.955583] bfe0: 00000020 76fa9d78 00000000 76e1e02c 60000010 0000001e 00000000 00000000
[  818.964692] Backtrace: 
[  818.969732] [<80858a8c>] (bcm2835_sdhost_dma_complete) from [<806e0084>] (vchan_complete+0x1e0/0x22c)
[  818.978947]  r8:00000122 r7:84dcbe20 r6:84dcbde8 r5:836d08bc r4:836d0884
[  818.987548] [<806dfea4>] (vchan_complete) from [<80126298>] (tasklet_action_common.constprop.5+0xe0/0xfc)
[  818.996081]  r10:ffffe000 r9:84dca000 r8:00000006 r7:00000000 r6:b77142d4 r5:81fe0364
[  819.003809]  r4:81665480
[  819.005678] [<801261b8>] (tasklet_action_common.constprop.5) from [<801262dc>] (tasklet_action+0x28/0x30)
[  819.016924]  r9:84dca000 r8:000000b0 r7:00000100 r6:00000006 r5:00000005 r4:80f03098
[  819.027371] [<801262b4>] (tasklet_action) from [<80101468>] (__do_softirq+0x198/0x474)
[  819.039510] [<801012d0>] (__do_softirq) from [<80126088>] (irq_exit+0xd8/0x120)
[  819.054595]  r10:10c5387d r9:84dca000 r8:60000010 r7:00000008 r6:00000000 r5:00000000
[  819.064093]  r4:ffffe000
[  819.069242] [<80125fb0>] (irq_exit) from [<8018822c>] (handle_domain_irq+0x70/0x88)
[  819.085216]  r5:00000000 r4:80e9110c
[  819.093258] [<801881bc>] (handle_domain_irq) from [<80101228>] (bcm2836_arm_irqchip_handle_irq+0x5c/0x60)
[  819.102428]  r7:84dcbf94 r6:ffffffff r5:60000013 r4:80100124
[  819.111640] [<801011cc>] (bcm2836_arm_irqchip_handle_irq) from [<80100aac>] (__irq_svc+0x4c/0x64)
[  819.125595] Exception stack(0x84dcbf60 to 0x84dcbfa8)
[  819.132431] bf60: 0000001e 76fa9d7c 3688a000 00000000 00000000 76fa9f48 0005dca0 000000ae
[  819.139186] bf80: 60000010 76e1e02c 10c5387d 00000000 84dcbfb0 84dcbfb0 80100120 80100124
[  819.145235] bfa0: 60000013 ffffffff
[  819.150937] Code: e3530000 0a000008 e5933000 e3a08000 (e5972028) 
[  819.165892] ---[ end trace 71381bd3ff2a2377 ]---
[  819.176066] Kernel panic - not syncing: Fatal exception in interrupt
[  819.186798] CPU3: stopping
[  819.188238] CPU: 3 PID: 396 Comm: kworker/3:0 Tainted: G      D W         5.15.21-v7+ #1523
[  819.191549] Hardware name: BCM2835
[  819.193149] Workqueue: events bpf_prog_free_deferred
[  819.195984] Backtrace: 
[  819.198043] [<80a3c57c>] (dump_backtrace) from [<80a3c7c4>] (show_stack+0x20/0x24)
[  819.203469]  r7:00000015 r6:80d18310 r5:00000080 r4:60000193
[  819.207568] [<80a3c7a4>] (show_stack) from [<80a40cfc>] (dump_stack_lvl+0x70/0x94)
[  819.211941] [<80a40c8c>] (dump_stack_lvl) from [<80a40d38>] (dump_stack+0x18/0x1c)
[  819.216445]  r7:00000015 r6:00000003 r5:80fe9344 r4:81050380
[  819.221099] [<80a40d20>] (dump_stack) from [<8010e6b0>] (do_handle_IPI+0x2dc/0x310)
[  819.227817] [<8010e3d4>] (do_handle_IPI) from [<8010e70c>] (ipi_handler+0x28/0x30)
[  819.233790]  r9:826a6000 r8:00000001 r7:00000015 r6:814e6340 r5:80fe9344 r4:814e3e00
[  819.238857] [<8010e6e4>] (ipi_handler) from [<8018e968>] (handle_percpu_devid_irq+0x8c/0x1d0)
[  819.243993] [<8018e8dc>] (handle_percpu_devid_irq) from [<80187c10>] (generic_handle_domain_irq+0x3c/0x4c)
[  819.251264]  r7:00000004 r6:814e3818 r5:80fe9464 r4:bc8110f0
[  819.255500] [<80187bd4>] (generic_handle_domain_irq) from [<8069a3f8>] (bcm2836_arm_irqchip_handle_ipi+0xa4/0xc4)
[  819.262625] [<8069a354>] (bcm2836_arm_irqchip_handle_ipi) from [<80188228>] (handle_domain_irq+0x6c/0x88)
[  819.268680]  r7:00000004 r6:00000000 r5:00000000 r4:80e9110c
[  819.271985] [<801881bc>] (handle_domain_irq) from [<80101228>] (bcm2836_arm_irqchip_handle_irq+0x5c/0x60)
[  819.277303]  r7:826a7d54 r6:ffffffff r5:80000013 r4:80118ffc
[  819.281200] [<801011cc>] (bcm2836_arm_irqchip_handle_irq) from [<80100aac>] (__irq_svc+0x4c/0x64)
[  819.286467] Exception stack(0x826a7d20 to 0x826a7d68)
[  819.290196] 7d20: a5b8e000 bc817000 00000001 80118fe4 80f05008 85cc6000 85cc6000 00000004
[  819.295388] 7d40: 00000001 81087718 b774ada8 826a7d9c 85cc5000 826a7d70 8010f734 80118ffc
[  819.301168] 7d60: 80000013 ffffffff
[  819.303686] [<8010f6c8>] (flush_tlb_kernel_range) from [<8030df10>] (__purge_vmap_area_lazy+0xc8/0x710)
[  819.309325]  r4:85cc5000
[  819.311481] [<8030de48>] (__purge_vmap_area_lazy) from [<8030eca0>] (_vm_unmap_aliases+0x134/0x170)
[  819.317243]  r10:b774ada8 r9:80f05188 r8:00000001 r7:00000004 r6:85cc6000 r5:85cc5000
[  819.321618]  r4:b774adc0
[  819.323315] [<8030eb6c>] (_vm_unmap_aliases) from [<80311bec>] (__vunmap+0x108/0x230)
[  819.327727]  r10:85cc6000 r9:b7769000 r8:00000001 r7:38e38e39 r6:85cc6000 r5:85c533c4
[  819.332721]  r4:85c53340
[  819.334672] [<80311ae4>] (__vunmap) from [<80311dbc>] (__vfree+0x50/0x90)
[  819.339051]  r10:b774ef05 r9:00000000 r8:b774ef00 r7:000000c0 r6:b774bb40 r5:85ccc5b8
[  819.343322]  r4:bc815000
[  819.344377] [<80311d6c>] (__vfree) from [<80311e60>] (vfree+0x64/0x7c)
[  819.348505]  r5:85ccc5b8 r4:bc815000
[  819.350958] [<80311dfc>] (vfree) from [<80254970>] (__bpf_prog_free+0x4c/0x50)
[  819.355460]  r5:85ccc5b8 r4:bc815000
[  819.357669] [<80254924>] (__bpf_prog_free) from [<80258928>] (bpf_prog_free_deferred+0xcc/0xe4)
[  819.362999]  r5:85ccc5b8 r4:85ccc400
[  819.365530] [<8025885c>] (bpf_prog_free_deferred) from [<8013c928>] (process_one_work+0x250/0x57c)
[  819.370991]  r7:000000c0 r6:b774bb40 r5:861c3500 r4:85ccc5b8
[  819.374738] [<8013c6d8>] (process_one_work) from [<8013ccb4>] (worker_thread+0x60/0x5c4)
[  819.378452]  r10:b774bb40 r9:80f03d00 r8:b774bb58 r7:00000008 r6:b774bb40 r5:861c3518
[  819.384070]  r4:861c3500
[  819.386003] [<8013cc54>] (worker_thread) from [<80144fe4>] (kthread+0x178/0x194)
[  819.391621]  r10:826a6000 r9:83711e74 r8:00000000 r7:861c3500 r6:8013cc54 r5:85c95c40
[  819.396180]  r4:85e09e80
[  819.397989] [<80144e6c>] (kthread) from [<801000d4>] (ret_from_fork+0x14/0x20)
[  819.403052] Exception stack(0x826a7fb0 to 0x826a7ff8)
[  819.406969] 7fa0:                                     00000000 00000000 00000000 00000000
[  819.412065] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  819.418422] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  819.423098]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80144e6c
[  819.428667]  r4:85c95c40
[  819.430805] CPU1: stopping
[  819.433232] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D W         5.15.21-v7+ #1523
[  819.437915] Hardware name: BCM2835
[  819.440834] Backtrace: 
[  819.442770] [<80a3c57c>] (dump_backtrace) from [<80a3c7c4>] (show_stack+0x20/0x24)
[  819.448337]  r7:00000015 r6:80d18310 r5:00000080 r4:60000193
[  819.452725] [<80a3c7a4>] (show_stack) from [<80a40cfc>] (dump_stack_lvl+0x70/0x94)
[  819.457689] [<80a40c8c>] (dump_stack_lvl) from [<80a40d38>] (dump_stack+0x18/0x1c)
[  819.462491]  r7:00000015 r6:00000001 r5:80fe9344 r4:81050380
[  819.466665] [<80a40d20>] (dump_stack) from [<8010e6b0>] (do_handle_IPI+0x2dc/0x310)
[  819.472359] [<8010e3d4>] (do_handle_IPI) from [<8010e70c>] (ipi_handler+0x28/0x30)
[  819.477547]  r9:8155c000 r8:00000001 r7:00000015 r6:814e6340 r5:80fe9344 r4:814e3e00
[  819.483269] [<8010e6e4>] (ipi_handler) from [<8018e968>] (handle_percpu_devid_irq+0x8c/0x1d0)
[  819.488677] [<8018e8dc>] (handle_percpu_devid_irq) from [<80187c10>] (generic_handle_domain_irq+0x3c/0x4c)
[  819.496270]  r7:00000004 r6:814e3818 r5:80fe9464 r4:bc8110d0
[  819.500486] [<80187bd4>] (generic_handle_domain_irq) from [<8069a3f8>] (bcm2836_arm_irqchip_handle_ipi+0xa4/0xc4)
[  819.506891] [<8069a354>] (bcm2836_arm_irqchip_handle_ipi) from [<80188228>] (handle_domain_irq+0x6c/0x88)
[  819.511952]  r7:00000004 r6:00000000 r5:00000000 r4:80e9110c
[  819.516717] [<801881bc>] (handle_domain_irq) from [<80101228>] (bcm2836_arm_irqchip_handle_irq+0x5c/0x60)
[  819.524094]  r7:8155df64 r6:ffffffff r5:60000013 r4:80108864
[  819.528862] [<801011cc>] (bcm2836_arm_irqchip_handle_irq) from [<80100aac>] (__irq_svc+0x4c/0x64)
[  819.534390] Exception stack(0x8155df30 to 0x8155df78)
[  819.537845] df20:                                     07484476 00000000 00000001 801190c0
[  819.543046] df40: ffffe000 00000002 80f05058 80f050a8 00000001 81037ca0 80d0eedc 8155df8c
[  819.549141] df60: 8155df90 8155df80 80108860 80108864 60000013 ffffffff
[  819.553262] [<8010881c>] (arch_cpu_idle) from [<80a4cc48>] (default_idle_call+0x4c/0xd8)
[  819.557874] [<80a4cbfc>] (default_idle_call) from [<8015a528>] (do_idle+0x114/0x164)
[  819.562597] [<8015a414>] (do_idle) from [<8015a89c>] (cpu_startup_entry+0x28/0x2c)
[  819.567792]  r10:00000000 r9:410fc075 r8:0000406a r7:8155dff8 r6:10c0387d r5:00000001
[  819.573316]  r4:00000096 r3:60000093
[  819.575363] [<8015a874>] (cpu_startup_entry) from [<8010ed30>] (secondary_start_kernel+0x168/0x174)
[  819.581612] [<8010ebc8>] (secondary_start_kernel) from [<001017f0>] (0x1017f0)
[  819.586188]  r5:00000055 r4:0157806a
[  819.589556] CPU2: stopping
[  819.599308] CPU: 2 PID: 113 Comm: systemd-journal Tainted: G      D W         5.15.21-v7+ #1523
[  819.605028] Hardware name: BCM2835
[  819.609717] Backtrace: 
[  819.614524] [<80a3c57c>] (dump_backtrace) from [<80a3c7c4>] (show_stack+0x20/0x24)
[  819.622910]  r7:00000015 r6:80d18310 r5:00000080 r4:60000193
[  819.627370] [<80a3c7a4>] (show_stack) from [<80a40cfc>] (dump_stack_lvl+0x70/0x94)
[  819.632232] [<80a40c8c>] (dump_stack_lvl) from [<80a40d38>] (dump_stack+0x18/0x1c)
[  819.636753]  r7:00000015 r6:00000002 r5:80fe9344 r4:81050380
[  819.639372] [<80a40d20>] (dump_stack) from [<8010e6b0>] (do_handle_IPI+0x2dc/0x310)
[  819.643541] [<8010e3d4>] (do_handle_IPI) from [<8010e70c>] (ipi_handler+0x28/0x30)
[  819.647891]  r9:00000000 r8:10c5387d r7:00000015 r6:814e6340 r5:80fe9344 r4:814e3e00
[  819.652074] [<8010e6e4>] (ipi_handler) from [<8018e968>] (handle_percpu_devid_irq+0x8c/0x1d0)
[  819.657656] [<8018e8dc>] (handle_percpu_devid_irq) from [<80187c10>] (generic_handle_domain_irq+0x3c/0x4c)
[  819.663279]  r7:00000004 r6:814e3818 r5:80fe9464 r4:bc8110e0
[  819.666365] [<80187bd4>] (generic_handle_domain_irq) from [<8069a3f8>] (bcm2836_arm_irqchip_handle_ipi+0xa4/0xc4)
[  819.673328] [<8069a354>] (bcm2836_arm_irqchip_handle_ipi) from [<80188228>] (handle_domain_irq+0x6c/0x88)
[  819.679223]  r7:00000004 r6:00000000 r5:00000000 r4:80e9110c
[  819.682248] [<801881bc>] (handle_domain_irq) from [<80101228>] (bcm2836_arm_irqchip_handle_irq+0x5c/0x60)
[  819.687540]  r7:10c5387d r6:ffffffff r5:80000010 r4:76ec725c
[  819.690936] [<801011cc>] (bcm2836_arm_irqchip_handle_irq) from [<80100dac>] (__irq_usr+0x4c/0x60)
[  819.696703] Exception stack(0x82ae1fb0 to 0x82ae1ff8)
[  819.700600] 1fa0:                                     fffffffe 000001ed 00000424 00000000
[  819.707103] 1fc0: 7e821990 00000000 00000001 00000027 7e81f158 00000000 00000009 7e81f0a4
[  819.711995] 1fe0: 004cddfc 7e81f064 004b27cc 76ec725c 80000010 ffffffff
[  819.719549] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Additional context

No response