Azure / Azure/azure-iot-sdk-node
AMQPWS/AMQP Connection Does Not Detect Disconnection in a Timely Manner
- Dominant language
- JavaScript
- Stars
- 268
- Forks
- 228
- PR merge metrics
- No merged PRs in 30d
Description
We are migrating our device connections from MQTT to AMQP over WebSockets (AMQPWS) to leverage features such as message abandon and completion. The devices are vehicles, and we have been conducting tests to evaluate this transition.
During testing, we observed that when a device experiences a temporary disconnection (e.g., passing through a tunnel) that lasts for approximately 5 minutes, the Azure IoT SDK does not immediately detect the network loss. Instead, the disconnection event is only triggered approximately 18 minutes after the initial loss of internet access.
We tested both AMQP and AMQPWS, and both exhibit the same behavior. This issue prevents us from releasing this feature, as the delayed disconnection detection could lead to significant operational issues.
### **Steps to Reproduce:**
- Connect a device using AMQPWS to Azure IoT Hub.
- Simulate a network disconnection lasting at least 2 minutes.
- Restore the internet connection.
- Observe that no disconnection event is triggered immediately upon reconnection.
- Notice that the SDK only detects the disconnection event approximately 18 minutes later.
### **One test with the retry policy disabled**
`rhea:events [connection-1] Connection got event: disconnected +18m`
### **One test with the retry policy enabled**
`rhea:events [connection-1] Connection got event: disconnected +22m`
### **Expected Behavior:**
The SDK should promptly detect a disconnection and attempt a reconnection as soon as network access is restored.
### **Observed Behavior:**
The disconnection event is only triggered approximately 18 minutes after the initial network loss.
### **Test Scenarios:**
Lab Environment: Device connected to a regular internet connection, manually disconnecting the network for at least 5 minutes.
Real-World Test: Device operating in a vehicle driving through the city, where natural disconnections occur (e.g., tunnels, poor signal areas). In this case, logs extracted after a full day of operation show the same delayed disconnection detection.
Local Test: Issue is not present when testing locally on a regular laptop.
### **Environment Details:**
- Architecture: 64-bit ARM
- Device OS: Ubuntu 18.04.6 LTS (Bionic Beaver)
- Application Environment: Running in a Docker container
- Docker Version: 20.10.21, build 20.10.21-0ubuntu1~18.04.3
- Container Base Image: node:20.10-alpine
### **Additional Information:**
- We have tested with and without the retry policy enabled, but the issue persists in both cases.
- The issue occurs regardless of whether the disconnection is intentional (lab test) or natural (real-world test in a vehicle).
Contributor guide
Research direction
No source files or tests are named. Start by reproducing the AMQPWS scenario with the retry policy both enabled and disabled, then inspect the rhea connection logs around network loss and restoration. Done means the SDK detects the disconnection promptly and attempts reconnection when network access returns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, docker, javascript, linux, node.js, ubuntu
- Domain
- backend, cloud, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100