mt7915e: Message timeout while waiting for mcu response

I'm seeing the following timeouts on a mt7621 board with mt7915e wifi:

[63464.928109] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout
[63485.407585] mt7915e 0000:02:00.0: Message 0000aded (seq 1) timeout
[63505.887090] mt7915e 0000:02:00.0: Message 00005aed (seq 2) timeout
[63526.366557] mt7915e 0000:02:00.0: Message 000094ed (seq 3) timeout
[63546.846045] mt7915e 0000:02:00.0: Message 000094ed (seq 4) timeout
[63567.325544] mt7915e 0000:02:00.0: Message 00005aed (seq 5) timeout
[63587.805033] mt7915e 0000:02:00.0: Message 00005aed (seq 6) timeout
[63608.284525] mt7915e 0000:02:00.0: Message 00005aed (seq 7) timeout
[63628.764011] mt7915e 0000:02:00.0: Message 000025ed (seq 8) timeout

Once those messages show up, wifi is unusable. A reboot is necessary to fix it.

The board is running openwrt 21.02 with the mt76 from c67df0d3130a51d79b558f0329c2ca289c73b16e and with the latest mt76 master (minus multi-bss and WED, plus "mt76: lock mt76_queue before access").

The issue only reproduces with lots (40+) of connected active clients sometimes only after 40 hour and later after boot. Unfortunately, I can also not directly reproduce it locally myself. I only have reports from multiple deployed accesspoints.

It appears as if the mcu is locked up or at least communication with the mcu is no longer possible. SO it appears to be either a problem with the firmware or a problem with the communication between driver and firmware. Any ideas or suggestions further debug or fix the problem would be appreciated.

Start with the mt7915e MCU timeout messages in the report and compare the driver and firmware communication paths at the point where they occur. Reproduce under the reported conditions—40+ active clients and extended uptime—using the cited mt76 revisions. Done means identifying and fixing the lockup or communication failure, with Wi-Fi remaining usable without a reboot.