Azure / Azure/iotedge

While offline, edgeHub fails after automatic renewal of certificates

Open
#7,321 14 comments 5 reactions 1 assignee Claimed by @vipeller View on GitHub
Dominant language
C#
Stars
1.5k
Forks
473
Avg merge
2d 17h
Merged PRs (30d)
7

Description

Our company have a large number of IoT devices that rely on the offline capabilities of IoT Edge.

We recently discovered that devices can run offline at a maximum of ~25 days.
The behaviour seems to be caused by the automatic renewal of device/workload certificates. The renewal interval can be specified by setting the edgeHub environment variable ServerCertificateRenewAfterInMs but maxes out at 25 days(int32.max).

When the certificate is renewed, the edgeHub is stopped and fails to start again if the device is offline. This causes both data loss and complete failure of downstream devices to run configured modules. The edgeHub does not recover when connectivity is restored.

In the documentation found [here](https://learn.microsoft.com/en-us/azure/iot-edge/offline-capabilities?view=iotedge-1.5) the following is stated: _"While disconnected from IoT Hub, the IoT Edge device, its deployed modules, and any downstream devices can operate indefinitely."_

## Expected Behavior
IoT Edge modules including the edge hub can operate indefinitely in offline mode

## Current Behavior
Edge Hub stops after being offline for ~25 days which causes dataloss at the devices

## Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
1. Configure an edge device to run with specified CA certificates by updating config.toml
`[edge_ca]`
`cert = "file:///etc/pki/tls/certs/.full-chain.ca.cert.pem"`
`pk = "file:///etc/pki/tls/private/.key.pem"`
3. For the edgeHub module set the environment variable **ServerCertificateRenewAfterInMs** to 60000 ms in order to enforce renewal every minute.
4. When the device twin is downloaded by the device, simulate an offline situation by disconnecting the device from the internet.
5. After a few minutes time, observe how the edgeHub fails to start after renewal of certificates.

## Context (Environment)

### Output of `iotedge check`

Click here

```

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
× aziot-identity-service package is up-to-date - Error
could not query https://aka.ms/latest-aziot-identity-service for latest available version
‼ host time is close to reference time - Warning
Could not query NTP server
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
× host can connect to and perform TLS handshake with iothub AMQP port - Error
Could not connect to .azure-devices.net : could not complete TLS handshake
× host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Error
Could not connect to .azure-devices.net : could not complete TLS handshake
× host can connect to and perform TLS handshake with iothub MQTT port - Error
Could not connect to .azure-devices.net : could not complete TLS handshake

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
× aziot-edge package is up-to-date - Error
Error while fetching latest versions of edge components: could not send HTTP request
√ container time is close to host time - OK
‼ DNS server - Warning
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
× Agent image is valid and can be pulled from upstream - Error
Failed to login to ta01iotcrd01.azurecr.io
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
× container on the default network can connect to upstream AMQP port - Error
Container on the default network could not connect to .azure-devices.net:5671
× container on the default network can connect to upstream HTTPS / WebSockets port - Error
Container on the default network could not connect to .azure-devices.net:443
× container on the default network can connect to upstream MQTT port - Error
Container on the default network could not connect to .azure-devices.net:8883
× container on the IoT Edge module network can connect to upstream AMQP port - Error
Container on the azure-iot-edge network could not connect to .azure-devices.net:5671
× container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error
Container on the azure-iot-edge network could not connect to .azure-devices.net:443
× container on the IoT Edge module network can connect to upstream MQTT port - Error
Container on the azure-iot-edge network could not connect to .azure-devices.net:8883
23 check(s) succeeded.
2 check(s) raised warnings. Re-run with --verbose for more details.
12 check(s) raised errors. Re-run with --verbose for more details.

```

### Device Information
* Host OS: CentOS 7 (But reproducible on other OS:s too)
* Architecture: amd64
* Container OS: Linux containers

### Runtime Versions
* aziot-edged: 1.4.20
* Edge Agent: 1.4.38
* Edge Hub: 1.4.38
* Docker/Moby: 20.10.25

## Logs

aziot-edged logs
[iotedge_system_logs.txt](https://github.com/user-attachments/files/16100719/iotedge_system_logs.txt)

edge-agent logs
[edgeAgent_logs.txt](https://github.com/user-attachments/files/16100711/edgeAgent_logs.txt)

edge-hub logs
[edgeHub_logs.txt](https://github.com/user-attachments/files/16100674/edgeHub_logs.txt)

## Additional Information
Logs supplied as files due to max character limit.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.