SLZB-06MU over TCP: bridge stops forwarding UART<->socket for 30-100 s, then NCP aborts with ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT (measurements from both sides)

What happened?

The adapter is an SMLIGHT SLZB-06MU used over TCP (ember driver). Every
few hours Zigbee2MQTT dies with:

Received ERROR from adapter, with code=ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT
Fatal error, status=ASH_NCP_FATAL_ERROR
z2m: Adapter disconnected, stopping

This is not a Zigbee2MQTT bug — I am filing it because three earlier reports
of the same failure were closed as stale for lack of data
(#29550, #23861, #29957), and I now have measurements from both sides that
identify where the frames are lost. Hopefully this helps anyone else on the
same combination, and the numbers may be useful to SMLIGHT.

Summary: the bridge periodically stops moving bytes between the EFR32 UART
and the TCP socket for tens of seconds, while its web server, SSE stream and
network stack stay fully responsive. The NCP, getting no ACKs, exhausts its
retry budget and aborts the link. Host-side ASH counters are perfectly clean —
zigbee-herdsman never misses a beat.

Device: SLZB-06MU, PCB revision 173
Core firmware: SLZB-OS v3.3.1 (stable) and v3.3.8.dev6 — identical behaviour
Radio firmware: EmberZNet 8.0.2 [GA] build 397, EZSP 14 (20250220, sw_flow)
Mode: LAN coordinator, socket tcp://<bridge>:6638, usePackets=false,
useMultiClients=false, ZB1 serial 115200, HW flow off
Host: Zigbee2MQTT 2.13.0 (zigbee-herdsman 10.8.0), adapter: ember, one
device on the network (a TS011F plug)

Symptom

Every few hours the bridge stops moving bytes between the EFR32 UART and the
TCP socket for tens of seconds. Everything else on the device keeps running.
When forwarding resumes, all frames the NCP retransmitted during the stall
arrive at the host in one burst, and the NCP has already given up:

ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT

Zigbee2MQTT then stops, the container restarts, and the link comes back.

Measurements

Two stalls captured with full instrumentation on both sides.

Stall #1 — 37.5 s

07:41:22.898  last normal exchange (DATA frm=3 -> ACK)
              ... 37.5 s of silence ...
07:42:00.348  <--- DATA (retransmit)      07:42:00.350  ---> ACK  (2 ms)
07:42:00.350  <--- DATA (retransmit)      07:42:00.351  ---> ACK
07:42:00.352  <--- DATA                   07:42:00.354  ---> ACK
07:42:00.357  <--- [FRAME type=ERROR]     ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT

Seven retransmissions, sent by the NCP hundreds of milliseconds apart, reached
the host glued together within 10 ms — they had been buffered somewhere on
the bridge and released at once.

Stall #2 — 100.7 s

02:45:40.329  last normal exchange
              ... 100.7 s of silence ...
02:47:21.083  burst of retransmissions, host ACKs each in 1–2 ms
02:47:21.102  <--- [FRAME type=ERROR]

During that 100 s the rest of the device was fully responsive:

• SSE to the Home Assistant integration kept streaming every 5 s with no
  gaps (02:46:00, :05, :10, :15, :20, :25, :30, :35, :40, :45 …);
• the web server answered HTTP normally (a /api2?action=5 poll at
  02:46:08 returned the device log);
• ICMP stayed at 3–25 ms with zero loss;
• /metrics reported free_heap = 129 unchanged, temp = 44.7.

The device log itself contains only two lines for the whole event:

02:47:21  SocketServer: [EFR32MG21] Client disconnected (connection lost), id: 0
02:48:19  SocketServer: [EFR32MG21] New client, id: 0, ip: <host>

The data is not corrupted — it is only late

Host-side ASH counters after each stall:

NAK frames  : RX=0, TX=0
CRC errors  : RX=0
Comm errors : RX=0
ACK timeouts: RX=0

NCP-side counters (ezspReadAndClearCounters, polled once per minute for
24 h — 1100+ samples): ASH_OVERFLOW_ERROR, ASH_FRAMING_ERROR,
ASH_OVERRUN_ERROR, ASH_XOFF — all zero, always.

Host ACK latency, measured from a packet capture on the host NIC:
median 0.9 ms, p95 1.5 ms, worst over 24 h 12.5 ms. The host is never
the slow party.

What we ruled out

• Network — no TCP retransmissions at all (7493 socket samples), no ICMP
  loss, direct L2 path, no ARP conflict.
• Host load — load_average peaked (2.2) during the cleanest hours and was
  0.2 when stalls happened.
• Memory — free_heap does not move by a single unit over days.
• Broadcast/multicast load — steady 176–216 packets per 10 s while response
  time varied fourfold.
• send() blocking on the socket — the stall lasts far longer than
  4 × 50 ms, and the client is never dropped (socket_uptime keeps growing
  until the ERROR).
• Zigbee activity — identical message rates in stalled and healthy windows.

Response-time degradation (average ICMP 8 ms at night vs 41 ms at midday, with
spikes to 800 ms) follows a daily curve and is not a predictor: stalls
happened while response time was perfectly normal.

What v3.3.8.dev6 changed, and what it did not

dev6 splits the old single Connection Task into SockSrvNet (prio 6) and
SockSrvRadio (prio 8) — visible both in the binary and in the device log
(Net task started / Radio task started). The stalls are unchanged.
dev6 additionally panicked once after 10 h 45 min (Reboot reason: PANIC),
which v3.3.1 never did for us.

Since data from the NCP piles up in the UART driver's RX buffer and is released
in one burst, SockSrvRadio is evidently not draining the UART during the
stall — while no other task on the device is affected.

Two observations from the binary

• No task is registered with the task watchdog. The ESP-IDF component is
  present in the image (esp_task_wdt_init, esp_task_wdt_add, the «Task
  watchdog got triggered» handler), but there are zero calls to
  esp_task_wdt_add from application code. A SocketServer stall is therefore
  invisible to the device itself — nothing recovers it and nothing logs it.
• The radio link is driven one byte at a time: uart_read_bytes and
  uart_write_bytes are each called from exactly one place — the Arduino
  HardwareSerial wrappers, with length = 1. At 115200 that is ~11.5k driver
  calls per second at full rate.

Questions

1. Is this a known defect, and is there a fix or a workaround (socket options,
   usePackets mode, UART buffer sizing, a different NCP build)?
2. Would you consider adding a watchdog to the SocketServer tasks? A stall of
   30–100 s with the rest of the firmware healthy is currently invisible to the
   device itself — nothing in its own log marks it.
3. Could you share the pin map / Arduino board variant for SLZB-06MU
   (W5500 SPI CS/MISO/MOSI/SCK/RST/IRQ, EFR32 UART TX/RX, radio reset)? You
   published slzb-06-firmware under GPLv3 for earlier models; the 06MU
   variant is not in espressif/arduino-esp32/variants.

Happy to supply raw captures: pcap of the Zigbee socket, per-frame ASH logs
from both stalls, NCP counter series, and the device syslog stream.

Reproduction environment

Instrumentation running continuously: tcpdump on port 6638, tcpdump on all
other bridge traffic, /proc/net/tcp sampling every 5 s, ICMP twice per second,
/metrics once per minute, the device's own syslog (UDP) to a collector, and
per-frame zh:ember:uart:ash debug from zigbee-herdsman.

What did you expect to happen?

The adapter link to survive, or at least the stall to be detectable before the
NCP gives up.

How to reproduce it (minimal and precise)

Run an SLZB-06MU (or, from the reports above, an SLZB-06M) as a LAN coordinator
with the ember driver over tcp://<bridge>:6638 and wait. With the
self-healing automation removed from our setup the rate is roughly one drop per
19 hours; when something restarts Zigbee2MQTT automatically after each drop, the
rate rises to ~20 per day, because the EZSP re-initialisation that follows a
restart is itself the heaviest load on the bridge.

Zigbee2MQTT version

2.13.0 (zigbee-herdsman 10.8.0)

Adapter firmware version

8.0.2 [GA] build 397, EZSP 14 (20250220, sw_flow build)

Adapter

SMLIGHT SLZB-06MU over TCP, adapter: ember, SLZB-OS v3.3.1 and v3.3.8.dev6
(identical behaviour)

Setup

Zigbee2MQTT in Docker, one device on this network (a TS011F plug), bridge on
PoE, direct L2 path to the host, no VLAN or Wi-Fi in between.

Debug log

Per-frame zh:ember:uart:ash logs, packet captures of the Zigbee socket, NCP
counter series and the bridge's own syslog stream are all available — say which
would help and I will attach them.

Start with the ember adapter and its zh:ember:uart:ash per-frame logging while reviewing the TCP connection on port 6638, usePackets and useMultiClients settings. Compare the supplied packet captures, ASH logs, NCP counters and bridge syslog around a stall. Done would require an identified cause and an agreed fix or workaround, but the report does not name a repository file or test.