MQTT client can permanently stop reconnecting after the broker becomes unreachable

What happened?

This issue is a follow-up on #31864 which hits multiple users periodically. Lacking experience in Typescript but having the drive to get this issue solved I tasked Claude with finding a minimalistic fix. Below is Claude's rationale with typical "LLMsplaining" which I tried to keep (somewhat) to the point.

I'll open a PR shortly for a fix, this issue is intended as followup + the how and the why.

When the MQTT broker becomes unreachable, Zigbee2MQTT can reach a state where it never reconnects, not
even after the broker returns. Only a restart recovers it. Zigbee2MQTT keeps running and keeps logging
Not connected to MQTT server! every 10 seconds, but makes no further connection attempt.

The cause is in the mqtt (MQTT.js) client, which latches into a state where its own auto-reconnect is
permanently disabled. Zigbee2MQTT does not notice: lib/mqtt.ts connects once (lib/controller.ts:169)
and then relies entirely on MQTT.js auto-reconnect, and its 10s connectionTimer only logs.

What did you expect to happen?

That Zigbee2MQTT keeps retrying until the broker is reachable again.

How to reproduce it (minimal and precise)

Real setup:

1. Run the MQTT broker on a different host than Zigbee2MQTT.
2. Let Zigbee2MQTT connect.
3. Make the broker unreachable without a TCP reset, so packets are dropped rather than refused:
   iptables -I INPUT -p tcp --dport 1883 -j DROP on the broker host, or drop the tunnel. A docker stop
   sends a RST and does not trigger this.
4. Wait a few minutes, then restore reachability.
5. Zigbee2MQTT never reconnects; the log only repeats Not connected to MQTT server!.

Standalone reproduction, no Zigbee hardware or broker needed. Exits 1 on the bug, 0 when reconnect works:

// Simulates a broker host that silently disappears (dead tunnel / frozen VM): the TCP socket
// stays open but nothing is answered, and socket.destroy() does not complete immediately
// because a write is still pending.
const net = require("node:net");
const mqttPacket = require("mqtt-packet");
const {connectAsync} = require("mqtt");

const PORT = 18830;
const ts = () => new Date().toISOString().substring(11, 23);
const log = (...a) => console.log(ts(), ...a);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

let blackhole = false;

const server = net.createServer((socket) => {
    const parser = mqttPacket.parser({protocolVersion: 4});
    socket.on("error", () => {});
    parser.on("error", () => {});
    parser.on("packet", (packet) => {
        if (blackhole) return;
        if (packet.cmd === "connect") socket.write(mqttPacket.generate({cmd: "connack", returnCode: 0, sessionPresent: false}));
        else if (packet.cmd === "subscribe")
            socket.write(mqttPacket.generate({cmd: "suback", messageId: packet.messageId, granted: packet.subscriptions.map(() => 0)}));
        else if (packet.cmd === "publish" && packet.qos === 1) socket.write(mqttPacket.generate({cmd: "puback", messageId: packet.messageId}));
        else if (packet.cmd === "pingreq") socket.write(mqttPacket.generate({cmd: "pingresp"}));
    });
    socket.on("data", (d) => parser.parse(d));
});

(async () => {
    await new Promise((r) => server.listen(PORT, "127.0.0.1", r));

    // keepalive shortened from zigbee2mqtt's default 60 to speed the test up
    const client = await connectAsync(`mqtt://127.0.0.1:${PORT}`, {keepalive: 5});
    let attempts = 0;
    client.on("error", (e) => log("MQTT error:", e.message));
    client.on("reconnect", () => log(`>> reconnect attempt #${++attempts}  connected=${client.connected}`));

    // zigbee2mqtt's connection check (lib/mqtt.ts), logs only
    setInterval(() => {
        if (!client.connected) {
            log(
                `Not connected to MQTT server!  connected=${client.connected} reconnecting=${client.reconnecting} ` +
                    `disconnecting=${client.disconnecting} disconnected=${client.disconnected} ` +
                    `reconnectTimer=${client.reconnectTimer ? "armed" : "null"} attempts=${attempts}`,
            );
        }
    }, 5000);

    await sleep(1000);

    // A pending write on a black-holed socket makes destroy() complete late. Simulated here so the
    // test is deterministic; on a real dead tunnel the kernel does this for you.
    const sock = client.stream;
    const origDestroy = sock.destroy.bind(sock);
    sock.destroy = (...args) => {
        setTimeout(() => origDestroy(...args), 1500);
        return sock;
    };

    log("### broker host disappears (no answers, socket stays open)");
    blackhole = true;

    await sleep(40000);
    log("### broker host is BACK");
    blackhole = false;

    await sleep(40000);
    log(`### RESULT: connected=${client.connected} attempts=${attempts} -- expected: reconnected`);
    process.exit(client.connected ? 0 : 1);
})();

Reproduced on mqtt@5.15.2 (shipped in the 2.14.1 image) and mqtt@5.16.0 (what ^5.15.2 resolves to
today), on Node 22.

Zigbee2MQTT version

2.14.1

Adapter firmware version

Not relevant — this is in the MQTT client, independent of the Zigbee adapter.

Adapter

Not relevant — this is in the MQTT client, independent of the Zigbee adapter.

Setup

Docker container running on a raspberry Pi, the MQTT broker is hosted externally and reachable via a WireGuard tunnel.

Device database.db entry

Not applicable.

Debug log

Output of the standalone reproduction above, on mqtt@5.16.0 / Node 22, repeated identical lines elided:

mqtt 5.16.0 / node v22.23.2
17:25:11.129 ### broker host disappears (no answers, socket stays open)
17:25:17.644 MQTT error: Keepalive timeout
17:25:18.651 >> reconnect attempt #1  connected=true
17:25:20.140 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:25:25.149 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:25:49.158 MQTT error: connack timeout
17:25:51.144 ### broker host is BACK
17:25:55.195 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
...
17:26:30.260 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:26:31.151 ### RESULT: connected=false attempts=1 -- expected: reconnected

Two lines matter:

• 17:25:18.651 — connected is still true, so _reconnect() takes the end() branch instead of
  reconnecting
• from 17:25:20 — disconnecting=true and reconnectTimer=null, and they stay that way. No further
  attempt is made, including after the broker returns at 17:25:51.

Notes

The latch belongs upstream in MQTT.js. Candidates there:

• _cleanUp(forced) should set this.connected = false synchronously instead of waiting for the
  stream's close; that alone closes the race
• _reconnect()'s this.end(() => this.connect()) branch leaves disconnecting latched whenever
  reconnecting is true; connect()'s reset condition does not cover that case
• connackTimer is a single shared field across overlapping streams, so a late close from an orphaned
  socket can clear the live stream's only timeout

mqtt@5.16.0 addresses none of these, and lib/mqtt.ts is unchanged on dev with the dependency still
at ^5.15.2, so a Zigbee2MQTT-side guard may be worthwhile regardless. A minimal one, tested against the
compiled dist/mqtt.js in the 2.14.1 image:

// in the existing 10s connectionTimer in lib/mqtt.ts, inside `if (!this.isConnected())`
if (this.client.disconnecting) {
    logger.warning("Forcing reconnect to MQTT server");

    this.client.disconnecting = false;
    this.client.reconnect();
}

disconnecting is the latch itself, so no extra state or counter is needed, and the branch is only
reached in a state where Zigbee2MQTT is already permanently disconnected.

Clearing the flag first is required. In the latched state disconnecting === true while disconnected
is still undefined, because end() never completed — it is parked on this.once('outgoingEmpty', …)
waiting for a QoS 1 ack that can no longer arrive (Zigbee2MQTT's retained bridge/state republish sits
in outgoing). reconnect() then hits

if (this.disconnecting && !this.disconnected) { this._deferredReconnect = f } else { f() }

and defers itself indefinitely, so calling reconnect() on its own is a no-op. I measured three variants
against the 2.14.1 image: plain reconnect() does not recover, disconnecting = false + reconnect()
does, and rebuilding the client does.

With the guard in place the 2.14.1 image reconnects once the broker returns, and the test suite passes at
100% coverage. Happy to open a PR if this direction is acceptable.

Start with lib/mqtt.ts, especially the existing 10-second connectionTimer, and compare its behavior with the initial connection in lib/controller.ts:169. Run the standalone MQTT reproduction in the issue and inspect the reported client state transitions. Done means Zigbee2MQTT resumes connection attempts and reconnects after the broker becomes reachable, with the test suite still passing.