openwrt / openwrt/mt76

mt76 crash issue, and proposed resolution

Open
#763 28 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
888
Forks
436
PR merge metrics
No merged PRs in 30d

Description

Greetings,

I've been chasing down an mt76 crash issue on a Belkin RT3200 for the past month.
It happens once or twice a day when many devices are roaming.

The stack traces were not always consistent, but here was one that I pulled:

<6>[34253.568878] br-vlan12: port 3(wl1-ap0.4096) entered forwarding state
<4>[34253.603584] wl1-ap0.4102 selects TX queue 0, but real number of TX queues is 0
<1>[34253.603618] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
<7>[34253.841853] Call trace:
<7>[34253.844290]  eth_type_trans+0x44/0x19c
<7>[34253.848032]  ieee80211_rx_list+0x1cc/0xbcc [mac80211]
<7>[34253.853100]  mt76_rx_complete+0x220/0x35c [mt76]
<7>[34253.857718]  mt76_rx_poll_complete+0x2c8/0x4f0 [mt76]
<7>[34253.862767]  mt76_dma_rx_poll+0x2a4/0x4f0 [mt76]
<7>[34253.867383]  __napi_poll+0x54/0x1b0
<7>[34253.870866]  napi_threaded_poll+0x84/0xe4
<7>[34253.874869]  kthread+0x11c/0x130
<7>[34253.878091]  ret_from_fork+0x10/0x20
<0>[34253.881664] Code: 91003880 f9006440 f9419020 f9400083 (f9400000)
<4>[34253.887749] ---[ end trace 85019ca9a65c2328 ]---

I found that after adding a potentially missing spin_lock in a mac_sta_poll( ) function, and moving the sta_remove to the sta_pre_rcu_remove with a synchronize_rcu at the end, I was able to get a consistent stack trace, so, that seemed like good progress.
Here is that stack trace:

<1>[  113.323540] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
<7>[  113.561515] Call trace:
<7>[  113.563953]  mt7615_mac_sta_remove+0x100/0x930 [mt7615_common]
<7>[  113.569783]  __mt76_sta_remove+0x68/0xe4 [mt76]
<7>[  113.574316]  mt76_sta_pre_rcu_remove+0x44/0x10c [mt76]
<7>[  113.579452]  ieee80211_find_sta_by_link_addrs+0x430/0x540 [mac80211]
<7>[  113.585829]  sta_info_destroy_addr_bss+0x38/0x70 [mac80211]
<7>[  113.591415]  ieee80211_color_change_finish+0x1278/0x1500 [mac80211]
<7>[  113.597694]  cfg80211_check_station_change+0x1384/0x4720 [cfg80211]
<7>[  113.603970]  genl_family_rcv_msg_doit+0xb4/0x110
<7>[  113.608587]  genl_rcv_msg+0xd0/0x1c0
<7>[  113.612156]  netlink_rcv_skb+0x58/0x120
<7>[  113.615986]  genl_rcv+0x34/0x50
<7>[  113.619121]  netlink_unicast+0x1f0/0x2ec
<7>[  113.623037]  netlink_sendmsg+0x19c/0x3d0
<7>[  113.626954]  ____sys_sendmsg+0x258/0x2a0
<7>[  113.630873]  ___sys_sendmsg+0x78/0xc0
<7>[  113.634528]  __sys_sendmsg+0x54/0xb0
<7>[  113.638096]  __arm64_sys_sendmsg+0x20/0x30
<7>[  113.642183]  invoke_syscall+0x44/0x110
<7>[  113.645927]  el0_svc_common.constprop.0+0x48/0xf0
<7>[  113.650623]  do_el0_svc+0x18/0x20
<7>[  113.653930]  el0_svc+0x14/0x50
<7>[  113.656979]  el0t_64_sync_handler+0xe0/0x110
<7>[  113.661240]  el0t_64_sync+0x158/0x15c
<0>[  113.664899] Code: eb02001f 540000e0 f94276a2 f9427aa1 (f9000441)
<4>[  113.670983] ---[ end trace 6e69b3da17b58560 ]---

After a bit more debugging, I believe I discovered the root cause.
mac80211 is calling sta_remove( ) with an already removed 80211_sta pointer. This happens when a wifi device rapidly connects and disconnects over a very short period. Five minutes after the rapid connect/disconnect event (probably the inactivity timeout), mac80211 then calls the sta_remove( ) function a 2nd time with the same 80211_sta pointer. The drv_priv portion of this 80211_sta structure is zeroed out and that's where things go terribly wrong.

Such as a kernel oops due to the poll_list/rc_list lists having an invalid dereference to 0. Or depending on the timing, if it gets past that portion, then idx 0 ends up being targeted, which is needed for the beacon/frame management. This causes all kinds of weird timeout errors in the log followed by a spectacular crash.

Here is a snippet of logs that shows what is happening:

Mar 16 14:01:25 [ 6907.783912] mt7915e start sta_add=0x00000000f53bf1c3 idx=2
Mar 16 14:01:25 [ 6907.795647] mt7915e start sta_remove=0x00000000f53bf1c3 idx=2
Mar 16 14:01:26 [ 6907.902339] mt7915e start sta_add=0x00000000f53bf1c3 idx=2
Mar 16 14:01:26 [ 6907.913964] mt7915e start sta_remove=0x00000000f53bf1c3 idx=2
...
Mar 16 14:07:10 [ 7251.911771] mt7915e start sta_remove=0x00000000f53bf1c3 idx=0 <--- oops

I believe the correct fix here is to check for if (!wcid->idx && !wcid->sta) return; as the first line in the sta_remove( ) function

@Fail-Safe has also been compiling various commits along the way to assist in verifying that this is fixed.

I am offering up a PR to fix this, and I hope we can get it so it meets your level of standards.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the mt76 and mt7615 sta_remove paths named in the stack traces, especially mt7615_mac_sta_remove and __mt76_sta_remove, and compare them with the reported repeated sta_remove sequence. Reproduce or inspect the rapid connect/disconnect case and verify that a second removal with zeroed drv_priv does not dereference invalid state or target idx 0. The fix is done when the reported oops no longer occurs under that sequence and relevant kernel tests or logs remain clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.