21.02 branch, seeing crashes and high Interrupt errors

Made a build on 4/15 from the 21.02 branch to test on a ZBT-WE3526 (MT7621 / 7612 / 7603 )

OpenWrt 21.02-SNAPSHOT, r16013-50f2f25d58

Many nice improvements over the 19.07.x releases, such as much stronger signals, less latency on both bands.

However, still seeing recurrences of 7603 crashes, and rapidly increasing /proc/interrupt errors.

Here are the crash logs for two instances:

[105622.833568] ------------[ cut here ]------------ [105622.838329] WARNING: CPU: 2 PID: 1487 at target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2021-04-11-bf45b30d/mt7603/mac.c:206 mt7603_filter_tx+0x16c/0x174 [mt7603e] [105622.853172] Modules linked in: pppoe ppp_async pppox ppp_generic mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc sch_cake nf_reject_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conncount iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat act_connmark nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb mmc_block mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci [105622.853386] xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common [105622.947785] CPU: 2 PID: 1487 Comm: hostapd Not tainted 5.4.111 #0 [105622.953936] Stack : 00000001 8007d738 806c0000 806beb24 80720000 806beaec 806bdc40 8ee639cc [105622.962359] 80860000 8fe5d268 80708d03 80651b4c 00000002 00000001 8ee63970 e3a64ee8 [105622.970779] 00000000 00000000 808a0000 00000000 30000000 0000015f 00000018 312e342e [105622.979197] 00000000 00001487 00000000 68203a6d 80000000 80720000 00000000 8eeb452c [105622.987615] 00000009 00000009 00000009 8eec0000 00000008 8034e8dc 00000008 80860008 [105622.996032] ... [105622.998557] Call Trace: [105623.001110] [<8000b64c>] show_stack+0x30/0x100 [105623.005643] [<805950b8>] dump_stack+0xa4/0xdc [105623.010091] [<8002bfe8>] __warn+0xc0/0x10c [105623.014263] [<8002c090>] warn_slowpath_fmt+0x5c/0xac [105623.019317] [<8eeb452c>] mt7603_filter_tx+0x16c/0x174 [mt7603e] [105623.025314] [<8eeb19d4>] mt7603_sta_remove+0x94/0xfc [mt7603e] [105623.031242] [<8eea2d44>] __mt76_sta_remove+0x78/0xec [mt76] [105623.036897] [<8eea2e90>] mt76_sta_state+0xd8/0x1e4 [mt76] [105623.042533] [<8e809ee4>] sta_set_sinfo+0xfac/0x102c [mac80211] [105623.048727] ---[ end trace 91b91b401c16c82b ]---

Related syslog entries for the abve:

Fri Apr 16 22:43:04 2021 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 4c:24:98:88:f0:0d Fri Apr 16 22:43:05 2021 daemon.info hostapd: wlan0: STA 4c:24:98:88:f0:0d IEEE 802.11: authenticated Fri Apr 16 22:43:05 2021 daemon.info hostapd: wlan0: STA 4c:24:98:88:f0:0d IEEE 802.11: associated (aid 1) Fri Apr 16 22:43:05 2021 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 4c:24:98:88:f0:0d Fri Apr 16 22:43:05 2021 daemon.info hostapd: wlan0: STA 4c:24:98:88:f0:0d WPA: pairwise key handshake completed (RSN) Fri Apr 16 22:43:33 2021 daemon.info hostapd: wlan0: STA d4:a3:3d:6e:98:da IEEE 802.11: authenticated Fri Apr 16 22:43:33 2021 daemon.info hostapd: wlan0: STA d4:a3:3d:6e:98:da IEEE 802.11: associated (aid 10) Fri Apr 16 22:43:33 2021 daemon.notice hostapd: wlan1: Prune association for d4:a3:3d:6e:98:da Fri Apr 16 22:43:33 2021 daemon.notice hostapd: wlan1: AP-STA-DISCONNECTED d4:a3:3d:6e:98:da Fri Apr 16 22:43:33 2021 daemon.notice hostapd: wlan0: AP-STA-CONNECTED d4:a3:3d:6e:98:da Fri Apr 16 22:43:33 2021 daemon.info hostapd: wlan0: STA d4:a3:3d:6e:98:da WPA: pairwise key handshake completed (RSN) Fri Apr 16 22:44:03 2021 daemon.info hostapd: wlan1: STA d4:a3:3d:6e:98:da IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE) Fri Apr 16 22:57:17 2021 daemon.info hostapd: wlan1: STA d4:a3:3d:6e:98:da IEEE 802.11: authenticated Fri Apr 16 22:57:17 2021 daemon.info hostapd: wlan1: STA d4:a3:3d:6e:98:da IEEE 802.11: associated (aid 2) Fri Apr 16 22:57:17 2021 daemon.notice hostapd: wlan0: Prune association for d4:a3:3d:6e:98:da Fri Apr 16 22:57:17 2021 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED d4:a3:3d:6e:98:da Fri Apr 16 22:57:17 2021 daemon.notice hostapd: wlan1: AP-STA-CONNECTED d4:a3:3d:6e:98:da Fri Apr 16 22:57:17 2021 daemon.info hostapd: wlan1: STA d4:a3:3d:6e:98:da WPA: pairwise key handshake completed (RSN) Fri Apr 16 22:57:19 2021 kern.warn kernel: [105622.833568] ------------[ cut here ]------------

The interrupt errors after 23hrs of uptime:

cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 8: 40895361 40895407 40895357 40895395 MIPS GIC Local 1 timer 9: 888302 0 0 0 MIPS GIC 63 IPI call 10: 0 2766758 0 0 MIPS GIC 64 IPI call 11: 0 0 2167887 0 MIPS GIC 65 IPI call 12: 0 0 0 2703685 MIPS GIC 66 IPI call 13: 12654030 0 0 0 MIPS GIC 67 IPI resched 14: 0 14345113 0 0 MIPS GIC 68 IPI resched 15: 0 0 6532256 0 MIPS GIC 69 IPI resched 16: 0 0 0 3889478 MIPS GIC 70 IPI resched 17: 0 0 0 0 MIPS GIC 19 1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2 19: 12 0 0 0 MIPS GIC 33 ttyS0 20: 0 0 0 0 MIPS GIC 27 1e130000.sdhci 21: 0 0 0 0 MIPS GIC 29 xhci-hcd:usb1 22: 11445452 0 0 0 MIPS GIC 10 1e100000.ethernet 25: 79 0 14273108 0 MIPS GIC 31 mt76x2e 26: 62 0 0 25687525 MIPS GIC 32 mt7603e 27: 0 0 0 0 1e000600.gpio 18 keys ERR: 2006

Start with mt7603/mac.c around line 206 and the mt7603_filter_tx entry point, then compare the supplied kernel trace and hostapd events with /proc/interrupts. Reproduce on the reported OpenWrt 21.02-SNAPSHOT setup if possible; done would require identifying and correcting the crash and interrupt-error behavior with evidence from comparable logs.