Azure / Azure/azure-iot-hub-node

Client.open() fails with "rhea: connection disconnected"

Open
#31 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3
Forks
18
PR merge metrics
No merged PRs in 30d

Description

## Description of the issue

All of a sudden, in one of our environments any attempt to invoke a Direct Method on any device in the fleet consistently failed.
Searching into the logs we found out that the `Client.open()` method of the SDK failed after ~4 minutes with the following error:

```bash
{"message": "rhea: connection disconnected", "code": ""}
```

Notes:
* First of all, it's weird that the `code` property in the error message is empty.
* Instances of `Client` are being created and destroyed at each attempt, a fresh instance is created at any new attempt.
* Restarting the application did fix the problem, so it seems that the SDK was left in a bad state after a failed attempt and creating new `Client` instances in the running application was not enough to get rid of it.
* The metrics showed no exhaustion of resources in any Cloud component of the solution.
* The Diagnostic Tool for the IoT Hub targeted by this integration had a warning: "IoT Hub service event detected - We have detected that your IoT Hub likely has had maintenance events in the past 24 hours which could have led to temporary disconnection of devices [..] or other service interruptions". This notice showed up some time after we had started tracking failures.

## References

This issue is also similar to [this one](https://github.com/Azure/azure-iot-sdk-node/issues/778) reported for `azure-iot-sdk-node` in 2020. Note [this comment](https://github.com/Azure/azure-iot-sdk-node/issues/778#issuecomment-623613317) from @anthonyvercolano:

> The problem where you see d2c continuing (likely after some sort of error recovery) but c2d no longer works is likely a bug on our part. Hard to reproduce, but I've seen it before. Could at its root level be caused by transient network issues OR the service moving processing between nodes.

This was somewhat our situation. We could consistently receive device-to-cloud (D2C) messages, but weren't able to invoke Direct Methods on the devices. The comment above mentions cloud-to-device (C2D) messages, but both Direct Methods and C2D messages requires the application to connect to the IoT Hub first to send either of them.

Thank you for your time and I appreciate any advice you could share!

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names Client.open() and the rhea connection error but gives no repository file or test entry point. Start by locating Client.open() and tracing how disconnected connections and the empty code are handled. Done means explaining or fixing recovery after a transient service or network interruption and covering the behavior with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.