intel / intel/ethernet-linux-ice

[bug] [6.6.0] crash while deferred detaching XDP

Open
#23 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
60
Forks
21
PR merge metrics
No merged PRs in 30d

Description

Crash info:

```crash
Kernel Version: 6.6.0-47.004-shopee
OS Version: 24.04
Call Trace:
Oops: 0000 [#1] SMP NOPTI
CPU: 33 PID: 33147 Comm: kworker/33:3 Kdump: loaded Tainted: G OE 6.6.0-47.004-shopee #47~24.04
Hardware name: Dell Inc. PowerEdge R650xs/0441XG, BIOS 1.13.2 12/19/2023
Workqueue: events bpf_link_put_deferred
RIP: 0010:ice_alloc_rx_bufs+0xad/0x1e0 [ice]
Code: 49 89 45 e0 66 45 39 7c 24 3a 0f 84 f7 00 00 00 31 c0 66 41 89 45 08 41 83 ee 01 0f 84 a7 00 00 00 41 0f b7 84 24 d8 00 00 00 <48> 83 7b 08 00 75 9d 31 f6 bf 20 28 06 00 65 8b 15 de cf 96 3f 66
RSP: 0018:ff5c38d08fd07c68 EFLAGS: 00010246
RAX: 0000000000000600 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000fff RDI: ff1de01484983a00
RBP: ff5c38d08fd07c98 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ff1de01484983a00
R13: 0000000000000000 R14: 0000000000000fff R15: 0000000000000000
FS: 0000000000000000(0000) GS:ff1de052ffa00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000f4ee26004 CR4: 0000000000771ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:

? show_regs+0x6d/0x80
? __die+0x24/0x80
? page_fault_oops+0x99/0x1b0
? do_user_addr_fault+0x2e8/0x630
? ice_sched_replay_q_bw+0x33/0x50 [ice]
? exc_page_fault+0x83/0x190
? asm_exc_page_fault+0x27/0x30
? ice_alloc_rx_bufs+0xad/0x1e0 [ice]
ice_vsi_cfg_rxq+0x258/0x320 [ice]
ice_vsi_cfg_rxqs+0x73/0xb0 [ice]
ice_vsi_cfg_lan+0x61/0xe0 [ice]
ice_xdp_setup_prog+0xb9/0x4d0 [ice]
? __pfx_ice_xdp+0x10/0x10 [ice]
ice_xdp+0x7e/0x90 [ice]
dev_xdp_install+0xde/0x140
bpf_xdp_link_release+0xb7/0x1e0
bpf_link_free+0x70/0x100
bpf_link_put_deferred+0x12/0x20
process_one_work+0x178/0x330
worker_thread+0x30d/0x450
? _raw_spin_lock_irqsave+0xe/0x20
? __pfx_worker_thread+0x10/0x10
kthread+0xda/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x47/0x70
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1b/0x30

Modules linked in: ip_set bridge xfrm_user xfrm_algo xt_addrtype qrtr uio_pci_generic uio vfio_pci vfio_pci_core vfio_iommu_type1 vfio cuse overlay 8021q garp mrp stp llc bonding tls cfg80211 xt_conntrack xt_multiport xt_tcpudp nft_compat nf_tables sch_fq_codel rdma_ucm(OE) ib_uverbs(OE) rdma_cm(OE) iw_cm(OE) ib_cm(OE) ib_core(OE) toa(O) nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 lp parport msr binfmt_misc nls_iso8859_1 intel_rapl_msr dell_wmi ledtrig_audio sparse_keymap video intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common ipmi_ssif i10nm_edac skx_edac_common nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass rapl intel_cstate intel_uncore dell_smbios dcdbas mgag200 dax_hmem cxl_acpi drm_shmem_helper cxl_core intel_th_gth wmi_bmof dell_wmi_descriptor mei_me isst_if_mbox_pci isst_if_mmio drm_kms_helper i2c_i801 isst_if_common i2c_smbus intel_th_pci acpi_ipmi i2c_algo_bit mei intel_vsec intel_pch_thermal intel_th ipmi_si ipmi_msghandler acpi_power_meter
mac_hid dm_multipath nvme_fabrics(O) nvme_core(O) mlx_compat(OE) fuse drm efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 xfs btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd ice megaraid_sas ahci tg3 xhci_pci xhci_pci_renesas libahci wmi [last unloaded: ipmi_devintf]
CR2: 0000000000000008
```

OS env info:

```bash
# uname -a
Linux XXX 6.6.0-47.004-shopee #47~24.04 SMP Fri Dec 20 16:05:43 +08 2024 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

# ethtool -i ens1f0np0
driver: ice
version: 6.6.0-47.004-shopee
firmware-version: 3.10 0x8000ad4a 1.3106.0
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# modinfo ice
filename: /lib/modules/6.6.0-47.004-shopee/kernel/drivers/net/ethernet/intel/ice/ice.ko
firmware: intel/ice/ddp/ice.pkg
license: GPL v2
description: Intel(R) Ethernet Connection E800 Series Linux Driver
author: Intel Corporation,
srcversion: 581F3EC258F8AB642E7603A
alias: pci:v00008086d00001888sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Esv*sd*bc*sc*i*
alias: pci:v00008086d0000124Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Csv*sd*bc*sc*i*
alias: pci:v00008086d0000189Asv*sd*bc*sc*i*
alias: pci:v00008086d00001899sv*sd*bc*sc*i*
alias: pci:v00008086d00001898sv*sd*bc*sc*i*
alias: pci:v00008086d00001897sv*sd*bc*sc*i*
alias: pci:v00008086d00001894sv*sd*bc*sc*i*
alias: pci:v00008086d00001893sv*sd*bc*sc*i*
alias: pci:v00008086d00001892sv*sd*bc*sc*i*
alias: pci:v00008086d00001891sv*sd*bc*sc*i*
alias: pci:v00008086d00001890sv*sd*bc*sc*i*
alias: pci:v00008086d0000188Esv*sd*bc*sc*i*
alias: pci:v00008086d0000188Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000188Csv*sd*bc*sc*i*
alias: pci:v00008086d0000188Bsv*sd*bc*sc*i*
alias: pci:v00008086d0000188Asv*sd*bc*sc*i*
alias: pci:v00008086d0000159Bsv*sd*bc*sc*i*
alias: pci:v00008086d0000159Asv*sd*bc*sc*i*
alias: pci:v00008086d00001599sv*sd*bc*sc*i*
alias: pci:v00008086d00001593sv*sd*bc*sc*i*
alias: pci:v00008086d00001592sv*sd*bc*sc*i*
alias: pci:v00008086d00001591sv*sd*bc*sc*i*
depends:
retpoline: Y
intree: Y
name: ice
vermagic: 6.6.0-47.004-shopee SMP mod_unload modversions
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: ...
sig_hashalgo: sha512
signature: ...
parm: debug:netif level (0=none,...,16=all) (int)
```

## Reproduce steps

I tried but failed to reproduce the issue.

Here're the reproduce steps:
1. attach XDP with driver mode
2. pin the XDP link to bpffs
3. stop the XDP agent (the XDP prog is alive because of pinning)
4. rm XDP link from bpffs and ethtool ens1f0np0 -G rx 2048 tx 2048 at the same time

Here're the steps to confirm that the XDP prog is detached by deferred way:
1. build latest [btrace](https://github.com/leonhwangprojects/btrace)
2. capture the function stack by `./btrace -k bpf_link_put -k ice_vsi_cfg_rxq -k ice_xdp --kfunc-kmods ice -v --output-stack --output-arg 'ring->next_to_use'`

Here're some output of `btrace`:

```bash
$ sudo ./btrace -k bpf_link_put -k ice_vsi_cfg_rxq --kfunc-kmods ice -v --output-stack --output-arg 'ring->next_to_use'
2025/03/18 14:08:36 Failed to get addr2line for udp_diag_init at 0xffffffffc1d79072: not found
bpf_link_put args=((struct bpf_link *)link=0xffff961a9f50a100) retval=(void) cpu=0 process=(0:swapper/0)
Func stack:
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d79072:udp_diag_init+0x62
0xffffffffbc4d0ec5:bpf_link_put+0x5 ; kernel/bpf/syscall.c:2926
0xffffffffbc6c17be:i_callback+0x1e ; fs/inode.c:250
0xffffffffbc3b3ed3:rcu_do_batch+0x1c3 ; kernel/rcu/tree.c:2189
0xffffffffbc3bb0e3:rcu_core+0x113 ; kernel/rcu/tree.c:2464
0xffffffffbc3bb3ee:rcu_core_si+0xe ; kernel/rcu/tree.c:2480
0xffffffffbc2f0735:handle_softirqs+0xe5 ; kernel/softirq.c:553
0xffffffffbc2f09e7:__irq_exit_rcu+0x77 ; kernel/softirq.c:587
0xffffffffbc2f0d3e:irq_exit_rcu+0xe ; kernel/softirq.c:651
0xffffffffbd299c12:sysvec_apic_timer_interrupt+0x92
0xffffffffbd400f4b:asm_sysvec_apic_timer_interrupt+0x1b ; arch/x86/include/asm/idtentry.h:649
0xffffffffbd29c3ed:cpu_idle_poll.isra.0+0x2d
0xffffffffbc35f545:do_idle+0x45 ; kernel/sched/idle.c:284
0xffffffffbc35f7fa:cpu_startup_entry+0x2a ; kernel/sched/idle.c:379
0xffffffffbd29d332:rest_init+0xc2
0xffffffffbe85a54e:parse_early_options+0xae
0xffffffffbe85a92f:start_kernel+0x34f ; init/main.c:1074
0xffffffffbe86de78:x86_64_start_reservations+0x18 ; arch/x86/kernel/head64.c:543
0xffffffffbe86dfbf:x86_64_start_kernel+0xbf ; arch/x86/kernel/head64.c:485
0xffffffffbc200274:secondary_startup_64_no_verify+0x18f ; arch/x86/kernel/head_64.S:449

2025/03/18 14:08:37 Failed to get addr2line for udp_diag_init at 0xffffffffc1d79072: not found
bpf_link_put args=((struct bpf_link *)link=0xffff961a0f63b540) retval=(void) cpu=0 process=(43720:kworker/0:0)
Func stack:
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d79072:udp_diag_init+0x62
0xffffffffbc4d0ec5:bpf_link_put+0x5 ; kernel/bpf/syscall.c:2926
0xffffffffbc6c17be:i_callback+0x1e ; fs/inode.c:250
0xffffffffbc3b3ed3:rcu_do_batch+0x1c3 ; kernel/rcu/tree.c:2189
0xffffffffbc3bb0e3:rcu_core+0x113 ; kernel/rcu/tree.c:2464
0xffffffffbc3bb3ee:rcu_core_si+0xe ; kernel/rcu/tree.c:2480
0xffffffffbc2f0735:handle_softirqs+0xe5 ; kernel/softirq.c:553
0xffffffffbd2b2ff0:__do_softirq+0x10
0xffffffffbc2f0421:do_softirq.part.0+0x41 ; kernel/softirq.c:454
0xffffffffbc2f0a88:__local_bh_enable_ip+0x58 ; arch/x86/include/asm/preempt.h:27
0xffffffffbd2b246d:_raw_spin_unlock_bh+0x1d ; kernel/locking/spinlock.c:211
0xffffffffbc4d16c5:bpf_link_free+0x45 ; kernel/bpf/syscall.c:2896
0xffffffffbc4d17a2:bpf_link_put_deferred+0x12 ; kernel/bpf/syscall.c:2920
0xffffffffbc3105b5:process_one_work+0x175 ; kernel/workqueue.c:2631
0xffffffffbc310d3d:worker_thread+0x30d ; kernel/workqueue.c:2698
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b

2025/03/18 14:08:37 Failed to get addr2line for udp_diag_init at 0xffffffffc1d79072: not found
bpf_link_put args=((struct bpf_link *)link=0xffff961a9f50a280) retval=(void) cpu=0 process=(15:ksoftirqd/0)
Func stack:
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc026fc00:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d79072:udp_diag_init+0x62
0xffffffffbc4d0ec5:bpf_link_put+0x5 ; kernel/bpf/syscall.c:2926
0xffffffffbc6c17be:i_callback+0x1e ; fs/inode.c:250
0xffffffffbc3b3ed3:rcu_do_batch+0x1c3 ; kernel/rcu/tree.c:2189
0xffffffffbc3bb0e3:rcu_core+0x113 ; kernel/rcu/tree.c:2464
0xffffffffbc3bb3ee:rcu_core_si+0xe ; kernel/rcu/tree.c:2480
0xffffffffbc2f0735:handle_softirqs+0xe5 ; kernel/softirq.c:553
0xffffffffbc2f0bec:run_ksoftirqd+0x3c ; kernel/softirq.c:925
0xffffffffbc327d17:smpboot_thread_fn+0xb7 ; kernel/smpboot.c:164
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b

2025/03/18 14:08:38 Failed to get addr2line for ice_xdp at 0xffffffffc07790d5: not found
ice_vsi_cfg_rxq args=((struct ice_rx_ring *)ring=0xffff961a1cf3c000) retval=(int)0 cpu=0 process=(43720:kworker/0:0)
Arg attrs: (u16)next_to_use=0x7ff/2047
Func stack:
0xffffffffc0270180:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc0270180:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d7b072:bpf_trampoline_45097288656+0x72
0xffffffffc0799215:ice_vsi_cfg_rxq+0x5
0xffffffffc0777b91:ice_vsi_cfg_lan+0x61
0xffffffffc0778ca9:ice_xdp_setup_prog+0xb9
0xffffffffc077914e:ice_xdp+0x7e
0xffffffffc1d7d047:+0x0
0xffffffffc07790d5:ice_xdp+0x5
0xffffffffbcf91917:bpf_xdp_link_release+0xb7 ; net/core/dev.c:9425
0xffffffffbc4d16ed:bpf_link_free+0x6d ; kernel/bpf/syscall.c:2899
0xffffffffbc4d17a2:bpf_link_put_deferred+0x12 ; kernel/bpf/syscall.c:2920
0xffffffffbc3105b5:process_one_work+0x175 ; kernel/workqueue.c:2631
0xffffffffbc310d3d:worker_thread+0x30d ; kernel/workqueue.c:2698
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b

2025/03/18 14:08:38 Failed to get addr2line for ice_xdp at 0xffffffffc07790d5: not found
ice_xdp args=((struct net_device *)dev=0xffff961a1ded6000, (struct netdev_bpf *)xdp=0xffffaaf1201c7d98) retval=(int)0 cpu=0 process=(43720:kworker/0:0)
Func stack:
0xffffffffc0270704:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc0270704:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d7d07a:+0x0
0xffffffffc07790d5:ice_xdp+0x5
0xffffffffbcf91917:bpf_xdp_link_release+0xb7 ; net/core/dev.c:9425
0xffffffffbc4d16ed:bpf_link_free+0x6d ; kernel/bpf/syscall.c:2899
0xffffffffbc4d17a2:bpf_link_put_deferred+0x12 ; kernel/bpf/syscall.c:2920
0xffffffffbc3105b5:process_one_work+0x175 ; kernel/workqueue.c:2631
0xffffffffbc310d3d:worker_thread+0x30d ; kernel/workqueue.c:2698
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b

2025/03/18 14:08:39 Failed to get addr2line for ice_xdp at 0xffffffffc07790d5: not found
ice_vsi_cfg_rxq args=((struct ice_rx_ring *)ring=0xffff961a1c973400) retval=(int)0 cpu=0 process=(60108:kworker/0:1)
Arg attrs: (u16)next_to_use=0x7ff/2047
Func stack:
0xffffffffc0270180:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc0270180:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d7b072:bpf_trampoline_45097288656+0x72
0xffffffffc0799215:ice_vsi_cfg_rxq+0x5
0xffffffffc0777b91:ice_vsi_cfg_lan+0x61
0xffffffffc0778ca9:ice_xdp_setup_prog+0xb9
0xffffffffc077914e:ice_xdp+0x7e
0xffffffffc1d7d047:+0x0
0xffffffffc07790d5:ice_xdp+0x5
0xffffffffbcf91917:bpf_xdp_link_release+0xb7 ; net/core/dev.c:9425
0xffffffffbc4d16ed:bpf_link_free+0x6d ; kernel/bpf/syscall.c:2899
0xffffffffbc4d17a2:bpf_link_put_deferred+0x12 ; kernel/bpf/syscall.c:2920
0xffffffffbc3105b5:process_one_work+0x175 ; kernel/workqueue.c:2631
0xffffffffbc310d3d:worker_thread+0x30d ; kernel/workqueue.c:2698
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b

2025/03/18 14:08:39 Failed to get addr2line for ice_xdp at 0xffffffffc07790d5: not found
ice_xdp args=((struct net_device *)dev=0xffff961a1cefb000, (struct netdev_bpf *)xdp=0xffffaaf10e1dfd98) retval=(int)0 cpu=0 process=(60108:kworker/0:1)
Func stack:
0xffffffffc0270704:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc0270704:fexit_fn+0x318 ; bpf/btrace.c:106
0xffffffffc1d7d07a:+0x0
0xffffffffc07790d5:ice_xdp+0x5
0xffffffffbcf91917:bpf_xdp_link_release+0xb7 ; net/core/dev.c:9425
0xffffffffbc4d16ed:bpf_link_free+0x6d ; kernel/bpf/syscall.c:2899
0xffffffffbc4d17a2:bpf_link_put_deferred+0x12 ; kernel/bpf/syscall.c:2920
0xffffffffbc3105b5:process_one_work+0x175 ; kernel/workqueue.c:2631
0xffffffffbc310d3d:worker_thread+0x30d ; kernel/workqueue.c:2698
0xffffffffbc31c167:kthread+0xd7 ; kernel/kthread.c:388
0xffffffffbc258004:ret_from_fork+0x44
0xffffffffbc20513b:ret_from_fork_asm+0x1b
```

From the output of `btrace`, we can confirm that the XDP progs are detached by deferred way.

Any idea to troubleshoot and resolve this crash issue?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.