rcu_sched detected stalls

Hi,

I have one AW7915-NPD and one AW7915-NP1 inside a clearfog gt 8k and I'm seeing some stalls/crashes usually during high traffic but can also happen constantly after some time. This also seems to happen with only one card as well.
Device becomes unresponsive and can only be recovered by removing power
I got the following log over serial

root@OpenWrt:/# [  184.045603] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  184.051575] rcu:     3-...0: (1 ticks this GP) idle=572/1/0x4000000000000000 softirq=5076/5076 fqs=935
[  184.060757]  (detected by 0, t=2102 jiffies, g=7101, q=25)
[  184.066272] Task dump for CPU 3:
[  184.069521] task:mt76-tx phy0    state:R  running task     stack:    0 pid: 1452 ppid:     2 flags:0x0000000a
[  184.079497] Call trace:
[  184.081972]  __switch_to+0x9c/0xfc
[  184.085419]  __mt76_worker_fn+0x0/0x100 [mt76]
[  186.025411] mt7915e 0001:01:00.0: Message 00005aed (seq 10) timeout
[  206.503270] mt7915e 0001:01:00.0: Message 000026ed (seq 11) timeout
[  247.089964] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  247.095932] rcu:     3-...0: (1 ticks this GP) idle=572/1/0x4000000000000000 softirq=5076/5076 fqs=3727
[  247.105201]  (detected by 0, t=8407 jiffies, g=7101, q=43)
[  247.110716] Task dump for CPU 3:
[  247.113965] task:mt76-tx phy0    state:R  running task     stack:    0 pid: 1452 ppid:     2 flags:0x0000000a
[  247.123940] Call trace:
[  247.126413]  __switch_to+0x9c/0xfc
[  247.129853]  __mt76_worker_fn+0x0/0x100 [mt76]
[  310.136842] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  310.142811] rcu:     3-...0: (1 ticks this GP) idle=572/1/0x4000000000000000 softirq=5076/5076 fqs=6244
[  310.152081]  (detected by 2, t=14713 jiffies, g=7101, q=178)
[  310.157771] Task dump for CPU 3:
[  310.161018] task:mt76-tx phy0    state:R  running task     stack:    0 pid: 1452 ppid:     2 flags:0x0000000a
[  310.170994] Call trace:
[  310.173467]  __switch_to+0x9c/0xfc
[  310.176910]  __mt76_worker_fn+0x0/0x100 [mt76]

Start by tracing __mt76_worker_fn and the mt7915e message-timeout path in the driver, then reproduce the RCU stalls under high traffic with one and two AW7915 cards. Use the serial log to correlate the stuck mt76-tx task with firmware timeouts. Done means the cause is identified and the driver no longer leaves the device unresponsive under the reported workload.