[Ember] Multi-hop source route causes deterministic unicast failures — EFR32MG24 / EmberZNet 8.0.2

What happened?

On an EFR32MG24 coordinator running EmberZNet 8.0.2 [GA] build 397 (EZSP 14), direct ZCL unicasts to several Philips Hue routers become reproducibly undeliverable after the NCP learns a source route containing one or more intermediate relays.

The most reproducible device is:

• Philips Hue 9290012573A / LCT001
• firmware 67.116.10, HW 3
• IEEE 0x0017880100ef050e
• NWK 8865

The reproducible sequence is:

fresh Zigbee2MQTT start
  sourceRoutes=0
  sourceRouteOverhead=0

→ first direct ZCL unicast succeeds

→ incoming route record:
  source=8865
  relayCount=1
  relayList=[18051]

→ NCP stores source route:
  Coordinator → 18051 → 8865
  sourceRouteOverhead=4

→ next direct ZCL unicast fails:
  ZIGBEE_DELIVERY_FAILED

Example from an instrumented run:

15:50:11 [MG24-PRE] attempt=1 sourceRoutes=0 overhead=0(OK)
15:50:11 [MG24-ROUTE-RECORD] source=8865 sourceEui=0x0017880100ef050e lastHopLqi=88 lastHopRssi=-78 relayCount=1 relayList=18051

15:50:15 [MG24-PRE] attempt=1 sourceRoutes=2 overhead=4(OK)

15:50:20 ZCL command 0x0017880100ef050e/11 genOnOff.on(...)
         failed (Delivery failed for '8865'.)

The lamp does not physically switch when this happens.

An APS group/multicast command to a group containing the same lamp continues to switch it reliably while direct unicasts fail.

Power-cycling the lamp results in a normal device announce with the same NWK address, so a stale NWK address has been ruled out.

I observed several different learned source routes to the same destination, all exhibiting the same failure:

Coordinator → 18051 → 8865
sourceRouteOverhead=4
FAILS

Coordinator → 5192 → 32090 → 8865
sourceRouteOverhead=6
FAILS

Coordinator → 41336 → 8865
sourceRouteOverhead=4
FAILS

The last route is particularly interesting because both hops have very good link quality:

Coordinator → 41336: LQI ~163
41336 → 8865:        LQI ~252

Therefore the failure is not limited to one weak or defective intermediate router.

A representative failure produced these NCP counters:

MAC_TX_UNICAST_SUCCESS=7
MAC_TX_UNICAST_RETRY=0
MAC_TX_UNICAST_FAILED=0

APS_DATA_TX_UNICAST_SUCCESS=2
APS_DATA_TX_UNICAST_FAILED=1

These counters are consistent with MAC transmissions from the coordinator succeeding while the end-to-end APS unicast fails. I do not interpret them as proving where the frame is ultimately discarded.

I also instrumented ezspIncomingRouteErrorHandler; no corresponding route-error callback was observed during the reproduced failure.

Decisive A/B test

For diagnosis I temporarily configured the NCP before network initialization with:

SOURCE_ROUTE_TABLE_SIZE = 0

No other Zigbee network parameter was changed:

• no re-pairing
• same PAN / Extended PAN ID
• same channel
• same coordinator IEEE
• same network key
• same devices
• same physical coordinator position

Incoming route records still arrive:

[MG24-ROUTE-RECORD] source=8865 relayCount=1 relayList=18051

but the NCP can no longer store a source route:

[MG24-PRE] sourceRoutes=0 overhead=0(OK)

Result:

• 10 consecutive direct ON/OFF unicasts to the affected lamp succeeded
• zero Delivery failed
• route records from the same lamp continued to arrive
• all other lamps tested afterwards also switched correctly

Example:

16:08:15 PRE sourceRoutes=0 overhead=0
16:08:15 ROUTE_RECORD source=8865 relayList=18051
16:08:17 PRE sourceRoutes=0 overhead=0
16:08:21 PRE sourceRoutes=0 overhead=0
16:08:23 PRE sourceRoutes=0 overhead=0
16:08:26 PRE sourceRoutes=0 overhead=0
16:08:31 PRE sourceRoutes=0 overhead=0
16:08:34 PRE sourceRoutes=0 overhead=0
16:08:37 PRE sourceRoutes=0 overhead=0
16:08:40 PRE sourceRoutes=0 overhead=0
16:08:43 PRE sourceRoutes=0 overhead=0

This is why I believe the failure is specifically associated with storing/using a learned multi-hop source route rather than basic RF reachability or reception of the route record itself.

Other lamps showing the same general failure pattern included LWO001 and LTW012.

What did you expect to happen?

A direct APS/ZCL unicast over a learned source route should be delivered when a valid path exists.

If a learned source route becomes unusable, I would expect it to be invalidated/repaired or route discovery to recover communication rather than subsequent direct unicasts remaining reproducibly undeliverable.

Group communication to the same destination demonstrates that the device itself remains reachable.

How to reproduce it (minimal and precise)

In my network this is reproducible as follows:

1. Start Zigbee2MQTT using the ember adapter and the default high-RAM concentrator configuration.
2. Restart Zigbee2MQTT so that no usable source route for NWK 8865 is stored.
3. Confirm:

   sourceRouteOverhead(8865) = 0
   

4. Send a direct ZCL ON/OFF command to NWK 8865.
5. The command succeeds.
6. An incoming route record is received:

   source=8865
   relayCount=1
   relayList=[18051]
   

7. The NCP stores:

   Coordinator → 18051 → 8865
   sourceRouteOverhead=4
   

8. Send another direct ZCL ON/OFF command to NWK 8865.
9. The command fails after ~5 seconds with:

   ZIGBEE_DELIVERY_FAILED
   

10. Send an APS group command containing the same lamp.
11. The lamp reacts normally.

This sequence has been reproduced multiple times.

As an A/B control, setting SOURCE_ROUTE_TABLE_SIZE=0 makes both direct commands succeed while the same incoming route records continue to arrive.

Zigbee2MQTT version

2.12.1 (Also reproduced with: Zigbee2MQTT 2.13.0 zigbee-herdsman 10.8.0)

Adapter firmware version

EmberZNet 8.0.2 [GA] build 397 EZSP 14

Adapter

Internal EFR32MG24 coordinator of a SMLIGHT SMHUB Nano MG24. (ember)

Setup

Plain Zigbee2MQTT installation running directly on a SMLIGHT SMHUB Nano MG24.

Mosquitto and Node-RED run locally on the SMHUB. Home Assistant runs on a separate machine and communicates via MQTT.

Hardware/software:

SMLIGHT SMHUB Nano
HW 0.98
SMHUB firmware 1.0.0
EFR32MG24 internal coordinator

Linux SMHUB 6.18.17-patch21 #1 Wed Mar 4 08:06:40 UTC 2026 riscv64 GNU/Linux

Zigbee2MQTT 2.12.1-1
zigbee-herdsman 10.6.1

Zigbee network:

channel: 11
PAN ID: 0xF106
27 devices
7 groups
~16 routers

The router population is mainly Philips/Signify Hue bulbs, including:

• LCT001
• LCT007
• LTG002
• LWO001
• LTW012

plus an OSRAM Plug 01 and several EnOcean PTM 215Z Green Power switches.

The network was migrated without re-pairing from a SMLIGHT SLZB-06MU.

The coordinator IEEE, PAN ID, Extended PAN ID, channel and network key were preserved.

The previous and current coordinators were placed in the same physical location.

The same Zigbee2MQTT 2.12.1 stack on the same SMHUB had previously operated this network through the external SLZB-06MU for approximately two weeks without these symptoms.

Device database.db entry

{"id":10,"type":"Router","ieeeAddr":"0x0017880100ef050e","nwkAddr":8865,"manufId":4107,"manufName":"Signify Netherlands B.V.","powerSource":"Mains (single phase)","modelId":"LCT001","epList":[11],"endpoints":{"11":{"profId":49246,"epId":11,"devId":528,"inClusterList":[0,3,4,5,6,8,4096,768,64515,64513],"outClusterList":[25],"clusters":{"genBasic":{"attributes":{"modelId":"LCT001","manufacturerName":"Signify Netherlands B.V.","powerSource":1,"zclVersion":1,"appVersion":2,"stackVersion":1,"hwVersion":3,"dateCode":"20250326","swBuildId":"67.116.10"}},"lightingColorCtrl":{"attributes":{"colorCapabilities":31,"colorTempPhysicalMin":153,"colorTempPhysicalMax":500}},"genOnOff":{"attributes":{"onOff":1}},"genLevelCtrl":{"attributes":{"currentLevel":252}}},"binds":[],"configuredReportings":[],"meta":{}}},"appVersion":2,"stackVersion":1,"hwVersion":3,"dateCode":"20250326","swBuildId":"67.116.10","zclVersion":1,"interviewCompleted":true,"interviewState":"SUCCESSFUL","meta":{"configured":"0.0.0"},"lastSeen":1787416520355}

Debug log

I can provide a trimmed debug log containing one complete reproduction:

• successful first direct unicast with sourceRouteOverhead=0
• incoming route record
• learned source route
• next direct unicast
• ZIGBEE_DELIVERY_FAILED
• matching neighbor / normal route / source-route / counter diagnostics

I can also provide the successful A/B run with SOURCE_ROUTE_TABLE_SIZE=0.

Notes

Tests already performed

Test	Result
Zigbee2MQTT 2.13.0 / zigbee-herdsman 10.8.0	Same failure
Low-RAM concentrator	Same failure
Removing ENABLE_ROUTE_DISCOVERY from DEFAULT_APS_OPTIONS	No fix
ezspSetConcentrator(false) + source-route discovery OFF	Incoming route records still resulted in stored source routes; failure remained
routingShortcutThreshold 1 → 7	No fix
TX power 20 → 5 dBm	No fix
Disable SMHUB 2.4 GHz Wi-Fi	No fix
Full MG24 NVM erase + restore network from coordinator backup	No fix
Power-cycle affected lamp	Valid device announce; problem returns
Power-cycle suspected intermediate router	A different source route is learned and fails
SOURCE_ROUTE_TABLE_SIZE=0	All tested lamps work

Current working workaround

The only workaround found so far is setting the source-route table size to zero before network initialization:

await this.emberSetEzspConfigValue(
    enums_2.EzspConfigId.SOURCE_ROUTE_TABLE_SIZE,
    0,
);

This currently requires patching zigbee-herdsman under node_modules, so the change is lost on updates.

I understand that disabling source-route storage may have undesirable consequences in larger/deeper Zigbee networks. I therefore consider this primarily an A/B diagnostic and temporary workaround, not a proposed default.

Hypothesis / open question

The A/B test strongly suggests that the failing condition is the outgoing multi-hop source-routed unicast.

I do not know which component is responsible.

Possibilities include:

• an EmberZNet 8.0.2 source-routing issue;
• interoperability between the generated source-routed frame and these Philips/Signify routers;
• an intermediate router failing to forward the source-routed frame;
• another Ember source-routing state issue not visible through the exposed route tables.

A packet capture comparing a successful non-source-routed frame with a failing source-routed frame would probably help distinguish these cases. I currently do not have a Zigbee sniffer available.

Possibly related to #29902, although that report describes a broader gradual routing/network degradation problem.

This issue is much more deterministic:

no multi-hop source route
→ direct unicast works

multi-hop source route learned
→ direct unicast fails

source-route storage disabled
→ direct unicast works

Start by tracing the ember adapter entry point emberSetEzspConfigValue, the source-route handling around ezspIncomingRouteErrorHandler, and the DEFAULT_APS_OPTIONS configuration mentioned in the report. Compare the successful no-source-route run with the failing learned-route run; packet captures or the supplied diagnostics should help identify whether the responsible layer is zigbee-herdsman, EmberZNet, or router interoperability. Done means the responsible layer and a validated fix or limitation are established.