Azure / Azure/iotedge

malformed DPS server response with TPM provisioning

Open
#6,247 6 comments 0 reactions 1 assignee Claimed by @jlian View on GitHub
customer-reported docs iotedge no-issue-activity
Dominant language
C#
Stars
1.5k
Forks
473
Avg merge
2d 17h
Merged PRs (30d)
7

Description

I have created a virtual machine with vTPM enabled in azure and followed the installation and TPM provisioning instructions as outlined in this article:
https://docs.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-tpm?view=iotedge-2020-11&tabs=virtual-machine%2Cubuntu#create-a-device-provisioning-service-enrollment

All steps executed/verified successfully up until applying the iotedge config and checking system logs. Output from the system logs is provided below. There appears to be some type of HTTP request error from the device provisioning service caused by `Connection reset by peer (os error 104)`, but no further indication of what might be wrong or how to fix it.

## Expected Behavior
The device should provision successfully

## Current Behavior
The identity service fails

## Context (Environment)
* Ubuntu 20.04, amd64
* Azure VM with vTPM feature enabled

### Output of `iotedge check`

Click here

```

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ tpmd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
could not load cert with ID "aziot-edged-trust-bundle"

Caused by:
parameter "id" has an invalid value
caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
SocketError - SocketErrorCode (TimedOut) : Operation timed out
One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
Container engine is not configured to rotate module logs which may cause it run out of disk space.
Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
Could not check current state of edgeAgent container
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
Could not check current state of edgeHub container
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
22 check(s) succeeded.
5 check(s) raised warnings. Re-run with --verbose for more details.
4 check(s) raised errors. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

```

### Runtime Versions
* aziot-edged [run `iotedge version`]: 1.2.8
* Edge Agent [image tag (e.g. 1.0.0)]: not installed
* Edge Hub [image tag (e.g. 1.0.0)]: not installed
* Docker/Moby [run `docker version`]: 20.10.14+azure-1

## Logs
The following log repeats continuously as the service attempts provisioning, fails, and restarts.

iotedge system logs

```

Apr 01 20:35:36 n3uron-longterm systemd[1]: Started Azure IoT Identity Service.
Apr 01 20:35:36 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:36Z [INFO] - Starting service...
Apr 01 20:35:36 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:36Z [INFO] - Version - 1.2.6
Apr 01 20:35:36 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:36Z [INFO] - Provisioning starting. Reason: Startup
Apr 01 20:35:36 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:36Z [INFO] - <-- GET /get_tpm_keys?api-version=2020-09-01 {"content-type": "application/json", "host": "tpmd.sock", "content-length": "2"}
Apr 01 20:35:36 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:36Z [INFO] - --> 200 {"content-type": "application/json"}
Apr 01 20:35:36 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:36Z [INFO] - <-- POST /import_auth_key?api-version=2020-09-01 {"content-type": "application/json", "host": "tpmd.sock", "content-length": "1274"}
Apr 01 20:35:37 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:37Z [INFO] - --> 200 {"content-type": "application/json"}
Apr 01 20:35:37 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:37Z [INFO] - <-- POST /sign_with_auth_key?api-version=2020-09-01 {"content-type": "application/json", "host": "tpmd.sock", "content-length": "131"}
Apr 01 20:35:37 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:37Z [INFO] - --> 200 {"content-type": "application/json"}
Apr 01 20:35:37 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:37Z [INFO] - <-- POST /sign_with_auth_key?api-version=2020-09-01 {"content-type": "application/json", "host": "tpmd.sock", "content-length": "131"}
Apr 01 20:35:37 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:37Z [INFO] - --> 200 {"content-type": "application/json"}
Apr 01 20:35:41 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:41Z [INFO] - Obtaining edge device provisioning data...
Apr 01 20:35:47 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:47Z [INFO] - <-- POST /sign_with_auth_key?api-version=2020-09-01 {"content-type": "application/json", "host": "tpmd.sock", "content-length": "131"}
Apr 01 20:35:47 n3uron-longterm aziot-tpmd[1073]: 2022-04-01T20:35:47Z [INFO] - --> 200 {"content-type": "application/json"}
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:47Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: DPS client error
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:47Z [ERR!] - service encountered an error
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:47Z [ERR!] - caused by: DPS client error
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:47Z [ERR!] - caused by: malformed DPS server response
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 2022-04-01T20:35:47Z [ERR!] - 0:
Apr 01 20:35:47 n3uron-longterm aziot-identityd[2206]: 1:
Apr 01 20:35:47 n3uron-longterm systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 20:35:47 n3uron-longterm systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.
Apr 01 20:35:52 n3uron-longterm systemd[1]: aziot-identityd.service: Scheduled restart job, restart counter is at 11.
Apr 01 20:35:52 n3uron-longterm systemd[1]: Stopped Azure IoT Identity Service.
Apr 01 20:35:52 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:52Z [WARN] - The daemon could not start up successfully: Could not retrieve device information
Apr 01 20:35:52 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:52Z [WARN] - caused by: HTTP request error
Apr 01 20:35:52 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:52Z [WARN] - caused by: connection error: Connection reset by peer (os error 104)
Apr 01 20:35:52 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:52Z [WARN] - Requesting device reprovision.
Apr 01 20:35:52 n3uron-longterm aziot-edged[1059]: 2022-04-01T20:35:52Z [WARN] - The reprovisioning operation failed
Apr 01 20:35:52 n3uron-longterm systemd[1]: Started Azure IoT Identity Service.

```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.