Azure / Azure/iotedge

Error Message unclear: Error scraping edgeHub metrics endpoint connection refused

Open
#6,296 7 comments 0 reactions 1 assignee Claimed by @and-rewsmith View on GitHub
customer-reported iotedge no-issue-activity
Dominant language
C#
Stars
1.5k
Forks
473
Avg merge
2d 17h
Merged PRs (30d)
7

Description

When the test CA certificates expires, the cloud telemetry stops. The edgeHub and other custom module (OPCPublisher) still run without issues and don't show any errors or warnings in the log.
The edgeAgent logs show the following error:
```

```
## Expected Behavior
If the CA certificate expires it should be shown in the logs or in the iotedge check command.

## Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
1. Setup IoT Edge device with Test Ca certificate
2. After certificate expires check the logs of the running modules

## Context (Environment)

### Output of `iotedge check`

Click here

```
Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
‼ daemon configurations up-to-date with config.toml - Warning
/etc/aziot/config.toml was modified after keyd's config
You must run 'aziotctl config apply' to update keyd's config with the latest config.toml
√ identityd config toml file specifies a valid hostname - OK
× aziot-identity-service package is up-to-date - Error
could not query https://raw.githubusercontent.com/Azure/azure-iotedge/master/latest-aziot-identity-service.json for latest available version
‼ host time is close to reference time - Warning
Could not query NTP server
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
‼ configuration up-to-date with config.toml - Warning
/etc/aziot/config.toml was modified after edged's config
You must run 'iotedge config apply' to update edged's config with the latest config.toml
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
‼ aziot-edge package is up-to-date - Warning
Error while fetching latest versions of edge components: could not send HTTP request
√ container time is close to host time - OK
‼ DNS server - Warning
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: container engine - Warning
Device is not using a production-supported container engine (moby-engine).
Please see https://aka.ms/iotedge-prod-checklist-moby for details.
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ Agent image is valid and can be pulled from upstream - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the default network can connect to upstream MQTT port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream MQTT port - OK
29 check(s) succeeded.
6 check(s) raised warnings. Re-run with --verbose for more details.
1 check(s) raised errors. Re-run with --verbose for more details.

```

### Device Information
* Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]: Ubuntu 18.04
* Architecture [e.g. amd64, arm32, arm64]: amd64
* Container OS [e.g. Linux containers, Windows containers]: Linux containers

### Runtime Versions
* aziot-edged [run `iotedge version`]: 1.2.5
* Edge Agent [image tag (e.g. 1.0.0)]: 1.2.7
* Edge Hub [image tag (e.g. 1.0.0)]: 1.2.7
* Docker/Moby [run `docker version`]: 20.10.11+azure-3

## Logs

edge-agent logs

```
<6> 2022-04-19 06:19:03.233 +00:00 [INF] - Error scraping endpoint http://172.18.0.4:9600/metrics - System.Net.Http.HttpRequestException: Connection refused
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Diagnostics/MetricsScraper.cs:line 102
```

## Additional Information
Please provide any additional information that may be helpful in understanding the issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.