TX Queue Tail Pointer Freeze Causing WPA2 4-Way Handshake Timeout

MT7663U USB WiFi adapter experiences a firmware-level bug where the TX queue tail pointer freezes, preventing EAPOL frames from being transmitted and causing persistent 4-way handshake timeouts after 15-60 minutes of operation.

Environment

Hardware:

• Chipset: MediaTek MT7663U (USB ID: 0e8d:7663)
• Interface: USB 2.0 High-Speed
• Router: OpenWrt 23.05-SNAPSHOT on Rockchip armv8

Software:

• Kernel: Linux 5.15.181 (aarch64)
• mt76 driver version: commit 1e336a8582dce2ef32ddd440d423e9afef961e71 (2024-04-03)
• Firmware: N9 Firmware Version 7663mp1827, Build Time: 20200512032642
• OpenWrt build: r24203-67f14479b8

Network Configuration:

• Mode: Station (STA) on 5GHz
• SSID: TP-Link_B065_5G
• Security: WPA2-PSK/CCMP
• Channel: 44 (5220 MHz)
• Bandwidth: HT20

Problem Description

Symptoms

1. After system boot, MT7663U connects successfully and operates normally for 15-60 minutes
2. After this period, connection enters continuous disconnect/reconnect loop
3. Authentication ✅ and Association ✅ succeed, but WPA2 4-way handshake consistently times out ❌
4. Pattern repeats indefinitely with exponential backoff (10s → 120s)
5. Only full system reboot or interface reset temporarily resolves the issue

Root Cause Analysis

TX Queue Tail Pointer Freeze:

The MT7663U firmware's TX queue tail pointer becomes permanently stuck, preventing packets from being dequeued even though the hardware has transmitted them.

Evidence from debugfs monitoring:

Initial state (after boot - working):
queue 0: hw-queued=0, head=112, tail=112  ← Healthy
 
After 15-60 minutes (failing):
Time    hw-queued  head   tail   Status
0s      24         211    187    ← tail FROZEN
2s      24         211    187    ← no movement
4s      24         211    187    ← STUCK
6s      24         211    187    ← STUCK
8s      24         211    187    ← 24 packets permanently stuck

Impact:

• TX queue appears full to the driver (head - tail = 24 packets)
• New EAPOL frames cannot be queued
• Client's Message 2/4 (SNonce + MIC) never reaches AP
• AP times out waiting for handshake response after 5 seconds
• Connection fails with "Reason: 15=4WAY_HANDSHAKE_TIMEOUT"

Timeline of Failure

Boot → 5 min:      Firmware OK, tail pointer advances normally
                   WPA2 handshake succeeds ✅
 
5 min → 15 min:    Tail pointer starts freezing intermittently
                   Some handshakes succeed, some fail
 
15 min → 60 min:   Depends on traffic load and optimizations
                   Higher traffic = faster degradation
 
60+ min:           Tail pointer permanently frozen
                   All handshakes fail ❌
                   Connection impossible

Reproduction Steps

1. Configure MT7663U as WiFi client (STA mode) on 5GHz network
2. Connect to WPA2-PSK secured AP
3. Wait 15-60 minutes with normal network activity
4. Monitor debugfs: watch -n1 cat /sys/kernel/debug/ieee80211/phy0/mt76/xmit-queues
5. Observe tail pointer freeze when queue 0 shows: hw-queued > 0 but tail never advances
6. Check logs: logread | grep "4WAY_HANDSHAKE_TIMEOUT"

Expected Behavior

• TX queue tail pointer should advance as packets are transmitted
• Queue should drain properly: head catches up to tail
• EAPOL frames should be transmitted successfully
• WPA2 handshake should complete normally throughout adapter lifetime

Actual Behavior

• TX queue tail pointer freezes at arbitrary value (e.g., 187, 11, 112)
• Packets remain permanently "queued" in firmware
• Driver sees full queue, blocks new transmissions
• EAPOL frames cannot be queued
• Handshake fails after 5 seconds with timeout

Logs

Successful Connection (first 5 minutes)

daemon.notice wpa_supplicant[3199]: phy0-sta0: Associated with ac:15:a2:8c:b0:64
daemon.notice wpa_supplicant[3199]: phy0-sta0: WPA: Key negotiation completed [PTK=CCMP GTK=CCMP]
daemon.notice wpa_supplicant[3199]: phy0-sta0: CTRL-EVENT-CONNECTED

Failed Connection (after 15+ minutes)

kern.info kernel: phy0-sta0: authenticate with ac:15:a2:8c:b0:64
kern.info kernel: phy0-sta0: authenticated
kern.info kernel: phy0-sta0: associate with ac:15:a2:8c:b0:64
kern.info kernel: phy0-sta0: associated
kern.info kernel: phy0-sta0: deauthenticated (Reason: 15=4WAY_HANDSHAKE_TIMEOUT)
daemon.notice wpa_supplicant[3199]: phy0-sta0: WPA: 4-Way Handshake failed - pre-shared key may be incorrect
daemon.notice wpa_supplicant[3199]: phy0-sta0: CTRL-EVENT-SSID-TEMP-DISABLED reason=WRONG_KEY

dmesg (no TX timeout errors visible after optimizations)

[  116.506394] mt7663u 2-1.2:1.2: N9 Firmware Version: 7663mp1827
[  126.895923] phy0-sta0: authenticate with ac:15:a2:8c:b0:64
[  126.949025] phy0-sta0: authenticated
[  126.990222] phy0-sta0: RX AssocResp from ac:15:a2:8c:b0:64
[  126.998701] phy0-sta0: associated
[  132.xxx] phy0-sta0: deauthenticated from ac:15:a2:8c:b0:64 (Reason: 15)
(Pattern repeats every 2 minutes indefinitely)

Workarounds Attempted

1. Disabled USB Autosuspend ⚠️ Partial Help

echo -1 > /sys/bus/usb/devices/2-1.2/power/autosuspend
echo on > /sys/bus/usb/devices/2-1.2/power/control

Result: Extended stable time from 15min → 60min, but bug still occurs

2. Reduced TX Queue Limits ⚠️ Partial Help

echo 0 > /sys/kernel/debug/ieee80211/phy0/aql_enable
echo 10 > /sys/kernel/debug/ieee80211/phy0/mt76/idle-timeout
iw phy phy0 set txq limit 512

Result: Smaller frozen queue (8-24 packets vs 101), but tail still freezes

3. Reduced Channel Bandwidth ⚠️ Minimal Help

uci set wireless.radio0.htmode='HT20'  # instead of HT40

Result: Slightly delayed freeze onset, problem persists

4. Interface Reset ✅ Temporary Fix

ip link set phy0-sta0 down && sleep 2 && ip link set phy0-sta0 up

Result: Clears TX queue, works for ~5-15 minutes then fails again

5. System Reboot ✅ Temporary Fix

Result: Fully resolves issue for 15-60 minutes then recurs

Start by reproducing the failure on the listed MT7663U setup and monitor /sys/kernel/debug/ieee80211/phy0/mt76/xmit-queues while checking the 4WAY_HANDSHAKE_TIMEOUT logs. Trace the mt76 TX queue and USB firmware interaction to determine why the tail stops advancing; done means the queue drains and WPA2 handshakes continue after extended operation.