intel / intel/ethernet-linux-ice
[bug] kernel crash due to ice driver NULL pointer deref during reload
- Dominant language
- C
- Stars
- 60
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
We are seeing a fairly aggressive crash in one part of our fleet, Relevant crash dump data shared below.
TLDR, it looks like the out-of-tree driver is missing this backport from 6.5: https://github.com/torvalds/linux/commit/b3e7b3a6ee92ab927f750a6b19615ce88ece808f ("ice: prevent NULL pointer deref during reload"). cc @aloktion @lczapnik
In our specific reproduction, we're using:
ice 1.13.7
kernel 6.1.92
E810-XXV with firmware 4.40 / 22.5.7
I flipped through the Intel Ethernet release notes for [Ethernet 29.1 (ice 1.14.9)](https://www.intel.com/content/www/us/en/content-details/822537/intel-ethernet-controller-products-release-notes.html), and I did see a brief mention of a null dereference fix, but it is unclear if this is the same one
`NULL pointer dereference happens in the ICE out-of-tree ICE driver. Determine if PF Reset is in progress before accessing the coalesce settings. `
Digging deeper, looking at even the most current source code here, I don't see the same `rtnl_lock()` [treatment in ice_load()](https://github.com/intel/ethernet-linux-ice/blame/main/src/ice_main.c#L7137) from upstream fix https://github.com/torvalds/linux/commit/b3e7b3a6ee92ab927f750a6b19615ce88ece808f, so I'm guessing that specific fixed issue was referring to something else?
Note, we're not using RHEL's kernel, but this crash signature/fix appears to match 1:1 for their report here: https://access.redhat.com/solutions/7097254
```
crash> bt
PID: 2717185 TASK: ffff8c093e11ce00 CPU: 107 COMMAND: "ethtool"
#0 [ffffa59339447878] machine_kexec at ffffffffa505725d
#1 [ffffa593394478d0] __crash_kexec at ffffffffa512617a
#2 [ffffa59339447990] crash_kexec at ffffffffa5127288
#3 [ffffa59339447998] oops_end at ffffffffa5029aaa
#4 [ffffa593394479b8] page_fault_oops at ffffffffa5062565
#5 [ffffa59339447a38] exc_page_fault at ffffffffa5880732
#6 [ffffa59339447a60] asm_exc_page_fault at ffffffffa5a00c02
[exception RIP: ice_get_channels+117]
RIP: ffffffffc09c3cc5 RSP: ffffa59339447b18 RFLAGS: 00010246
RAX: ffff8a0fbdd4c828 RBX: ffff8c080137a280 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8c080137a288 RDI: 0000000000000080
RBP: ffff8a0f202da000 R8: ffff8a0f21a401e0 R9: ffff8a0fbdd4ca28
R10: ffff8a0f2385e028 R11: 0000000000000002 R12: ffff8a0f202da000
R13: ffff8c080137a280 R14: 0000000000000011 R15: ffffa59339447ba0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffa59339447b18] channels_prepare_data at ffffffffa56c93a1
#8 [ffffa59339447b30] ethnl_default_doit at ffffffffa56c46de
#9 [ffffa59339447b88] genl_family_rcv_msg_doit at ffffffffa56b856c
#10 [ffffa59339447c20] genl_rcv_msg at ffffffffa56b86e9
#11 [ffffa59339447ca8] netlink_rcv_skb at ffffffffa56b72ae
#12 [ffffa59339447d18] genl_rcv at ffffffffa56b7b14
#13 [ffffa59339447d28] netlink_unicast at ffffffffa56b67d1
#14 [ffffa59339447d68] netlink_sendmsg at ffffffffa56b6ac0
#15 [ffffa59339447de0] __sock_sendmsg at ffffffffa56225ef
#16 [ffffa59339447df8] __sys_sendto at ffffffffa5623ba0
#17 [ffffa59339447f30] __x64_sys_sendto at ffffffffa5623c35
#18 [ffffa59339447f38] do_syscall_64 at ffffffffa587cff3
#19 [ffffa59339447f50] entry_SYSCALL_64_after_hwframe at ffffffffa5a00126
RIP: 00007f334c73fcbb RSP: 00007ffca7a041c8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 000055e95457d340 RCX: 00007f334c73fcbb
RDX: 0000000000000024 RSI: 000055e95457d3b0 RDI: 0000000000000003
RBP: 000055e952c475a0 R8: 00007f334ca034c0 R9: 000000000000000c
R10: 0000000000000000 R11: 0000000000000246 R12: 000055e952c47980
R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffca7a043d8
ORIG_RAX: 000000000000002c CS: 0033 SS: 002b
```
Scheduled reset is being done by kworker on CPU 90
```
crash> bt 1662614
PID: 1662614 TASK: ffff8a0f2015ce00 CPU: 90 COMMAND: "kworker/90:0"
#0 [ffffa5935220fa68] __schedule at ffffffffa588876c
#1 [ffffa5935220fac8] schedule at ffffffffa5888e83
#2 [ffffa5935220fae0] schedule_timeout at ffffffffa588eaae
#3 [ffffa5935220fb58] wait_for_completion at ffffffffa58897c1
#4 [ffffa5935220fba8] __wait_rcu_gp at ffffffffa50ea609
#5 [ffffa5935220fbe0] synchronize_rcu at ffffffffa50f42d8
#6 [ffffa5935220fc28] ice_free_q_vector at ffffffffc0993138 [ice]
#7 [ffffa5935220fc50] ice_vsi_free_q_vectors at ffffffffc0993a30 [ice]
#8 [ffffa5935220fc68] ice_vsi_decfg at ffffffffc0997f19 [ice]
#9 [ffffa5935220fc80] ice_vsi_rebuild at ffffffffc0998ce5 [ice]
#10 [ffffa5935220fcf8] ice_vsi_rebuild_by_type at ffffffffc0967176 [ice]
#11 [ffffa5935220fd38] ice_rebuild at ffffffffc096f089 [ice]
#12 [ffffa5935220fdb0] ice_do_reset at ffffffffc096f884 [ice]
#13 [ffffa5935220fdd8] ice_service_task at ffffffffc09703cb [ice]
#14 [ffffa5935220fe80] process_one_work at ffffffffa509032f
#15 [ffffa5935220fec0] worker_thread at ffffffffa509050d
#16 [ffffa5935220ff18] kthread at ffffffffa5097a89
#17 [ffffa5935220ff50] ret_from_fork at ffffffffa50018ff
```
"ethtool" is invoked by systemd
```
crash> ps -p 2717185
PID: 0 TASK: ffffffffa6a11a40 CPU: 0 COMMAND: "swapper/0"
PID: 1 TASK: ffff8a0f0dc40000 CPU: 53 COMMAND: "systemd"
PID: 9492 TASK: ffff8b0efa1aa700 CPU: 103 COMMAND: "python3"
PID: 2717185 TASK: ffff8c093e11ce00 CPU: 107 COMMAND: "ethtool"
```
Contributor guide
Assessment
This issue has not been assessed yet.