Azure / Azure/iotedge

Memory Leak in EdgeHub module

Open
#6,997 8 comments 0 reactions 1 assignee Claimed by @vipeller View on GitHub
bug customer-reported iotedge no-issue-activity
Dominant language
C#
Stars
1.5k
Forks
473
Avg merge
2d 17h
Merged PRs (30d)
7

Description

## Expected Behavior
Memory utilization by EdgeHub module should be constant

## Current Behavior
Memory utilization by EdgeHub module is increasing

## Steps to Reproduce
1. Run EdgeHub module for 2-3 days
2. Observe memory utilization of the EdgeHub module
3. Restart EdgeHub module and check memory utilization

## Context (Environment)
### Output of `iotedge check`

damen@damen:~$ sudo iotedge check
[sudo] password for damen:

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
‼ aziot-identity-service package is up-to-date - Warning
Installed aziot-identity-service package has version 1.4.1 but 1.4.3 is the latest stable version available.
Please see https://aka.ms/aziot-update-runtime for update instructions.
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
‼ aziot-edge package is up-to-date - Warning
Installed IoT Edge daemon has version 1.4.3 but 1.4.9 is the latest stable version available.
Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ container time is close to host time - OK
√ DNS server - OK
‼ production readiness: logs policy - Warning
Container engine is not configured to rotate module logs which may cause it run out of disk space.
Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
You can ignore this warning if you are setting log policy per module in the Edge deployment.
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
32 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

### Device Information
* Host OS : Ubuntu 20.04
* Architecture : amd64
* Container OS : Linux containers

### Runtime Versions
* aziot-edged [run `iotedge version`]: 1.4.3
* Edge Agent [image tag (e.g. 1.0.0)]: mcr.microsoft.com/azureiotedge-agent:1.4.2-linux-amd64
* Edge Hub [image tag (e.g. 1.0.0)]: mcr.microsoft.com/azureiotedge-hub:1.4.2-linux-amd64
* Docker/Moby [run `docker version`]: 20.10.18+azure-2

## Additional Information
10-20 MB increasing at every 4 hours
Below are the memory utilization be Edge Hub
```
******************************************
* Mon Mar 13 16:00:01 UTC 2023 *
******************************************
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
737343ad5d99 devicemonitor 1.11% 1.906MiB / 500MiB 0.38% 27.4kB / 30.1kB 0B / 0B 3
3d3b2f853eb8 edgeHub 0.04% 163.4MiB / 500MiB 32.68% 26.6MB / 32.6MB 4.1kB / 2.45GB 25
d3cc99067218 devicemanagement 1.11% 2.23MiB / 500MiB 0.45% 0B / 0B 0B / 0B 3
c6025fa13fcc edgeAgent 0.00% 51.72MiB / 500MiB 10.34% 561kB / 600kB 8.19kB / 291kB 18
******************************************
* Mon Mar 13 20:00:01 UTC 2023 *
******************************************
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
737343ad5d99 devicemonitor 1.11% 1.91MiB / 500MiB 0.38% 55.4kB / 64.3kB 0B / 0B 3
3d3b2f853eb8 edgeHub 0.15% 182.7MiB / 500MiB 36.54% 38.7MB / 49.9MB 4.1kB / 3.31GB 25
d3cc99067218 devicemanagement 1.14% 2.223MiB / 500MiB 0.44% 0B / 0B 0B / 0B 3
c6025fa13fcc edgeAgent 0.00% 50.85MiB / 500MiB 10.17% 1.21MB / 1.32MB 8.19kB / 537kB 17
******************************************
* Tue Mar 14 00:01:01 UTC 2023 *
******************************************
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
737343ad5d99 devicemonitor 1.10% 1.914MiB / 500MiB 0.38% 78.5kB / 92.2kB 0B / 0B 3
3d3b2f853eb8 edgeHub 0.04% 197.6MiB / 500MiB 39.52% 48.3MB / 63.8MB 4.1kB / 3.99GB 24
d3cc99067218 devicemanagement 1.16% 2.227MiB / 500MiB 0.45% 0B / 0B 0B / 0B 3
c6025fa13fcc edgeAgent 1.37% 51.69MiB / 500MiB 10.34% 1.72MB / 1.9MB 8.19kB / 733kB 18
******************************************
* Tue Mar 14 04:01:01 UTC 2023 *
******************************************
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
737343ad5d99 devicemonitor 1.11% 1.922MiB / 500MiB 0.38% 101kB / 120kB 0B / 0B 3
3d3b2f853eb8 edgeHub 0.06% 211.9MiB / 500MiB 42.38% 58MB / 77.8MB 4.1kB / 4.68GB 26
d3cc99067218 devicemanagement 1.07% 2.23MiB / 500MiB 0.45% 0B / 0B 0B / 0B 3
c6025fa13fcc edgeAgent 0.00% 51.9MiB / 500MiB 10.38% 2.23MB / 2.47MB 8.19kB / 930kB 18
******************************************
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.