Unable to handle kernel paging request at virtual address 64662f72

Open
#4,730 11 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
18/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
c, linux
Domain
embedded-iot, operating-systems

Research direction

Start with the dmesg trace and the USB disconnect path named in it: usb_remove_ep_devs, device_del, and klist_del, after the usbip-host device disconnects. Investigate why endpoint removal reports the missing 'power' sysfs group and then reaches the invalid address. Done means reproducing the failure or a focused regression test shows USB disconnect completes without a warning or kernel oops.

Written by the indexing model from the issue text.

Description

Describe the bug
A kernel oops happens. Afterwards the rpi is unresponsive, hangs on operations like lsusb and soon afterwards even SSH access is no longer possible.

To reproduce
Sadly no easy way to reproduce however it occurs somewhat regularly (ca. once a week)

Expected behaviour
The kernel oops should not happen

Actual behaviour
The kernel oops happened

System

  • Which model of Raspberry Pi? Pi4B+ 8GB
  • Which OS and version? Raspberry Pi reference 2021-05-07 dcfd74d7d1fa293065ac6d565711e9ff891fe2b8, stage2, Raspbian GNU/Linux 10 (buster)
  • Which firmware version? 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea (clean) (release) (start_x)
  • Which kernel version? Linux <hostname> 5.10.63-v7l+ #1457 SMP Tue Sep 28 11:26:14 BST 2021 armv7l GNU/Linux

Logs

dmesg output with kernel oops 
[279800.385401] usbip-host 1-1.2: stub up
[279800.449989] usbip-host 1-1.2: usb_set_interface done: inf 0 alt 0
[279801.523799] usbip-host 1-1.1: stub up
[279821.313386] usbip-host 1-1.1: urb completion with non-zero status -71
[279821.314679] usbip-host 1-1.1: unlinked by a call to usb_unlink_urb()
[279821.392012] usbip-host 1-1.1: USB disconnect, device number 69
[279821.392060] ------------[ cut here ]------------
[279821.392093] WARNING: CPU: 0 PID: 14781 at fs/sysfs/group.c:281 sysfs_remove_group+0xa8/0xac
[279821.392108] sysfs group 'power' not found for kobject 'ep_03'
[279821.392122] Modules linked in: sha256_generic cfg80211 rfkill 8021q garp stp llc ftdi_sio usbserial vc4 cec v3d drm_kms_helper gpu_sched drm bcm2835_codec(C) v4l2_mem2mem drm_panel_orientation_quirks snd_soc_core snd_compress bcm2835_v4l2(C) bcm2835_isp(C) snd_pcm_dmaengine bcm2835_mmal_vchiq(C) raspberrypi_hwmon videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) videodev snd_pcm mc snd_timer vc_sm_cma(C) snd syscopyarea sysfillrect sysimgblt fb_sys_fops backlight rpivid_mem nvmem_rmem uio_pdrv_genirq uio usbip_host usbip_core ip_tables x_tables ipv6
[279821.392762] CPU: 0 PID: 14781 Comm: kworker/0:1 Tainted: G         C        5.10.63-v7l+ #1457
[279821.392771] Hardware name: BCM2711
[279821.392793] Workqueue: usb_hub_wq hub_event
[279821.392808] Backtrace: 
[279821.392833] [<c0b835a4>] (dump_backtrace) from [<c0b83938>] (show_stack+0x20/0x24)
[279821.392847]  r7:ffffffff r6:00000000 r5:60000013 r4:c12e6b3c
[279821.392863] [<c0b83918>] (show_stack) from [<c0b87d24>] (dump_stack+0xcc/0xf8)
[279821.392881] [<c0b87c58>] (dump_stack) from [<c0220b10>] (__warn+0xfc/0x114)
[279821.392895]  r10:00000001 r9:00000009 r8:c04f811c r7:00000119 r6:00000009 r5:c04f811c
[279821.392905]  r4:c0e48434 r3:c1205094
[279821.392919] [<c0220a14>] (__warn) from [<c0b840dc>] (warn_slowpath_fmt+0xa4/0xd8)
[279821.392930]  r7:00000119 r6:c0e48434 r5:c1205048 r4:c0e484dc
[279821.392946] [<c0b8403c>] (warn_slowpath_fmt) from [<c04f811c>] (sysfs_remove_group+0xa8/0xac)
[279821.392959]  r9:c2399000 r8:c3321820 r7:c1205048 r6:c424e608 r5:c0c4d9e4 r4:00000000
[279821.392978] [<c04f8074>] (sysfs_remove_group) from [<c0859698>] (dpm_sysfs_remove+0x58/0x6c)
[279821.392989]  r6:c3ffe988 r5:c424e64c r4:c424e608
[279821.393004] [<c0859640>] (dpm_sysfs_remove) from [<c0849964>] (device_del+0xc0/0x420)
[279821.393014]  r5:c424e64c r4:c424e608
[279821.393026] [<c08498a4>] (device_del) from [<c0849ce0>] (device_unregister+0x1c/0x30)
[279821.393038]  r10:00000001 r9:c2399000 r8:c3319600 r7:c3321800 r6:c3ffe988 r5:00000002
[279821.393047]  r4:c424e608
[279821.393061] [<c0849cc4>] (device_unregister) from [<c0904944>] (usb_remove_ep_devs+0x2c/0x38)
[279821.393070]  r5:00000002 r4:c4cc2fb8
[279821.393085] [<c0904918>] (usb_remove_ep_devs) from [<c08fc430>] (remove_intf_ep_devs+0x50/0x6c)
[279821.393094]  r5:00000002 r4:00000070
[279821.393109] [<c08fc3e0>] (remove_intf_ep_devs) from [<c08fd0cc>] (usb_disable_device+0x90/0x134)
[279821.393120]  r7:c3321800 r6:00000000 r5:00000002 r4:c3f66000
[279821.393136] [<c08fd03c>] (usb_disable_device) from [<c08f36e0>] (usb_disconnect+0xcc/0x248)
[279821.393148]  r9:c2399000 r8:c3319600 r7:c3f66078 r6:c3f660bc r5:c2399000 r4:c3f66000
[279821.393165] [<c08f3614>] (usb_disconnect) from [<c08f5864>] (hub_event+0xbc4/0x1308)
[279821.393177]  r10:00000001 r9:c2398f00 r8:00000100 r7:00000001 r6:c22e2000 r5:c2399000
[279821.393186]  r4:00000000
[279821.393204] [<c08f4ca0>] (hub_event) from [<c023db68>] (process_one_work+0x250/0x5a0)
[279821.393216]  r10:00000000 r9:00000000 r8:00000000 r7:efef7800 r6:efef26c0 r5:c4d70080
[279821.393224]  r4:c2398f00
[279821.393239] [<c023d918>] (process_one_work) from [<c023df18>] (worker_thread+0x60/0x5c4)
[279821.393252]  r10:efef26c0 r9:c1203d00 r8:efef26d8 r7:00000008 r6:efef26c0 r5:c4d70094
[279821.393260]  r4:c4d70080
[279821.393275] [<c023deb8>] (worker_thread) from [<c02459f8>] (kthread+0x170/0x174)
[279821.393288]  r10:c43e5e74 r9:c4d70080 r8:c023deb8 r7:c4232000 r6:00000000 r5:c3ec3680
[279821.393296]  r4:c358bd40
[279821.393311] [<c0245888>] (kthread) from [<c02000ec>] (ret_from_fork+0x14/0x28)
[279821.393321] Exception stack(0xc4233fb0 to 0xc4233ff8)
[279821.393333] 3fa0:                                     00000000 00000000 00000000 00000000
[279821.393345] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[279821.393356] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[279821.393369]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0245888
[279821.393378]  r4:c3ec3680
[279821.393412] ---[ end trace 4cc37ef73f243be9 ]---
[279821.393432] 8<--- cut here ---
[279821.396710] Unable to handle kernel paging request at virtual address 64662f72
[279821.404193] pgd = e830858d
[279821.407057] [64662f72] *pgd=80000000005003, *pmd=00000000
[279821.412853] Internal error: Oops: 205 [#1] SMP ARM
[279821.417744] Modules linked in: sha256_generic cfg80211 rfkill 8021q garp stp llc ftdi_sio usbserial vc4 cec v3d drm_kms_helper gpu_sched drm bcm2835_codec(C) v4l2_mem2mem drm_panel_orientation_quirks snd_soc_core snd_compress bcm2835_v4l2(C) bcm2835_isp(C) snd_pcm_dmaengine bcm2835_mmal_vchiq(C) raspberrypi_hwmon videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) videodev snd_pcm mc snd_timer vc_sm_cma(C) snd syscopyarea sysfillrect sysimgblt fb_sys_fops backlight rpivid_mem nvmem_rmem uio_pdrv_genirq uio usbip_host usbip_core ip_tables x_tables ipv6
[279821.471494] CPU: 0 PID: 14781 Comm: kworker/0:1 Tainted: G        WC        5.10.63-v7l+ #1457
[279821.480204] Hardware name: BCM2711
[279821.483712] Workqueue: usb_hub_wq hub_event
[279821.487999] PC is at klist_put+0x20/0xa4
[279821.492015] LR is at klist_del+0x14/0x18
[279821.496030] pc : [<c07721d0>]    lr : [<c0772268>]    psr: a0000013
[279821.502391] sp : c4233d10  ip : c4233d30  fp : c4233d2c
[279821.507710] r10: 00000001  r9 : c2399000  r8 : c3321820
[279821.513028] r7 : c3331e94  r6 : 00000001  r5 : c424e64c  r4 : 64662f62
[279821.519650] r3 : f6c6a3a1  r2 : f6c6a3a1  r1 : 00000001  r0 : 64662f62
[279821.526274] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[279821.533508] Control: 30c5383d  Table: 0403ec00  DAC: fffffffd
[279821.539349] Process kworker/0:1 (pid: 14781, stack limit = 0xcb6bffde)
[279821.545971] Stack: (0xc4233d10 to 0xc4234000)
[279821.550423] 3d00:                                     c424e608 c424e64c c3ffe988 c1205048
[279821.558703] 3d20: c4233d3c c4233d30 c0772268 c07721bc c4233d8c c4233d40 c0849978 c0772260
[279821.566982] 3d40: c4233d74 00000000 c020bc28 c020caec c4233d84 c4233d60 c020bc28 f6c6a3a1
[279821.575260] 3d60: c1205048 c424e608 00000002 c3ffe988 c3321800 c3319600 c2399000 00000001
[279821.583539] 3d80: c4233da4 c4233d90 c0849ce0 c08498b0 c4cc2fb8 00000002 c4233dbc c4233da8
[279821.591818] 3da0: c0904944 c0849cd0 00000070 00000002 c4233ddc c4233dc0 c08fc430 c0904924
[279821.600096] 3dc0: c3f66000 00000002 00000000 c3321800 c4233e04 c4233de0 c08fd0cc c08fc3ec
[279821.608374] 3de0: c3f66000 c2399000 c3f660bc c3f66078 c3319600 c2399000 c4233e3c c4233e08
[279821.616652] 3e00: c08f36e0 c08fd048 c4233e3c c4233e18 c08ef5ec 00000000 c2399000 c22e2000
[279821.624930] 3e20: 00000001 00000100 c2398f00 00000001 c4233ef4 c4233e40 c08f5864 c08f3620
[279821.633209] 3e40: c4233ea2 00000000 c032f050 c033027c c1097ac0 00000000 efef2ac0 c0eb83f8
[279821.641488] 3e60: c12050c8 00000000 00000001 c2398c20 c1205048 c22e20bc c23991dc c2398e40
[279821.649766] 3e80: c22e2000 c2398c00 c2399008 c23991dc c2398e38 c18d2000 c2398e00 c2398e34
[279821.658044] 3ea0: 00010100 c1095c10 c023db54 efef26c0 c023df18 00000000 00000000 00000000
[279821.666322] 3ec0: c4233ef4 f6c6a3a1 c032f050 c2398f00 c4d70080 efef26c0 efef7800 00000000
[279821.674600] 3ee0: 00000000 00000000 c4233f34 c4233ef8 c023db68 c08f4cac c032f30c c020cb1c
[279821.682878] 3f00: efef26c0 efef26c0 efef26c0 c4d70080 c4d70094 efef26c0 00000008 efef26d8
[279821.691156] 3f20: c1203d00 efef26c0 c4233f74 c4233f38 c023df18 c023d924 c0b938d4 c0e221fc
[279821.699434] 3f40: c133d1de ffffe000 c4233f74 c358bd40 c3ec3680 00000000 c4232000 c023deb8
[279821.707712] 3f60: c4d70080 c43e5e74 c4233fac c4233f78 c02459f8 c023dec4 c358bd64 c358bd64
[279821.715990] 3f80: ffffe000 c3ec3680 c0245888 00000000 00000000 00000000 00000000 00000000
[279821.724268] 3fa0: 00000000 c4233fb0 c02000ec c0245894 00000000 00000000 00000000 00000000
[279821.732545] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[279821.740823] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[279821.749097] Backtrace: 
[279821.751647] [<c07721b0>] (klist_put) from [<c0772268>] (klist_del+0x14/0x18)
[279821.758794]  r7:c1205048 r6:c3ffe988 r5:c424e64c r4:c424e608
[279821.764553] [<c0772254>] (klist_del) from [<c0849978>] (device_del+0xd4/0x420)
[279821.771876] [<c08498a4>] (device_del) from [<c0849ce0>] (device_unregister+0x1c/0x30)
[279821.779808]  r10:00000001 r9:c2399000 r8:c3319600 r7:c3321800 r6:c3ffe988 r5:00000002
[279821.787735]  r4:c424e608
[279821.790365] [<c0849cc4>] (device_unregister) from [<c0904944>] (usb_remove_ep_devs+0x2c/0x38)
[279821.798989]  r5:00000002 r4:c4cc2fb8
[279821.802662] [<c0904918>] (usb_remove_ep_devs) from [<c08fc430>] (remove_intf_ep_devs+0x50/0x6c)
[279821.811459]  r5:00000002 r4:00000070
[279821.815132] [<c08fc3e0>] (remove_intf_ep_devs) from [<c08fd0cc>] (usb_disable_device+0x90/0x134)
[279821.824018]  r7:c3321800 r6:00000000 r5:00000002 r4:c3f66000
[279821.829777] [<c08fd03c>] (usb_disable_device) from [<c08f36e0>] (usb_disconnect+0xcc/0x248)
[279821.838230]  r9:c2399000 r8:c3319600 r7:c3f66078 r6:c3f660bc r5:c2399000 r4:c3f66000
[279821.846078] [<c08f3614>] (usb_disconnect) from [<c08f5864>] (hub_event+0xbc4/0x1308)
[279821.853922]  r10:00000001 r9:c2398f00 r8:00000100 r7:00000001 r6:c22e2000 r5:c2399000
[279821.861849]  r4:00000000
[279821.864483] [<c08f4ca0>] (hub_event) from [<c023db68>] (process_one_work+0x250/0x5a0)
[279821.872414]  r10:00000000 r9:00000000 r8:00000000 r7:efef7800 r6:efef26c0 r5:c4d70080
[279821.880340]  r4:c2398f00
[279821.882970] [<c023d918>] (process_one_work) from [<c023df18>] (worker_thread+0x60/0x5c4)
[279821.891162]  r10:efef26c0 r9:c1203d00 r8:efef26d8 r7:00000008 r6:efef26c0 r5:c4d70094
[279821.899089]  r4:c4d70080
[279821.901720] [<c023deb8>] (worker_thread) from [<c02459f8>] (kthread+0x170/0x174)
[279821.909216]  r10:c43e5e74 r9:c4d70080 r8:c023deb8 r7:c4232000 r6:00000000 r5:c3ec3680
[279821.917143]  r4:c358bd40
[279821.919774] [<c0245888>] (kthread) from [<c02000ec>] (ret_from_fork+0x14/0x28)
[279821.927093] Exception stack(0xc4233fb0 to 0xc4233ff8)
[279821.932240] 3fa0:                                     00000000 00000000 00000000 00000000
[279821.940517] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[279821.948794] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[279821.955506]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0245888
[279821.963439]  r4:c3ec3680
[279821.966072] Code: e1a07000 e1a06001 e3c44001 e1a00004 (e5945010) 
[279821.972820] ---[ end trace 4cc37ef73f243bea ]---
[279821.980930] usbip-host 1-1.1: submit_urb error, -19
[279821.986367] usbip-host 1-1.1: lock for reset
[281110.483949] usbip-host 1-1.2: recv a header, -110
[281110.584378] usbip-host 1-1.2: reset full-speed USB device number 68 using xhci_hcd
[281110.723035] usbip-host 1-1.2: device reset

Additional context
I included a few kernel messages before and after the oops as it seems to be related to USB devices.

Dominant language
C
Stars
13.2k
Forks
5.5k
Avg merge
2d 21h
Merged PRs (30d)
21

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from raspberrypi/linux

All issues in raspberrypi/linux

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.