Azure / Azure/iotedge

Change in deployment fails due to corrupt JSON in container createOptions due to a fault JSON-merge

Open
#7,489 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.5k
Forks
473
Avg merge
2d 17h
Merged PRs (30d)
7

Description

## Expected Behavior
When changing from deployment A to B the edgeAgent try and merge (on JSON-level) the old deployment with the new. The expected behavior is that this merge shall succeed so edgeAgent can configure the modules correctly.

## Current Behavior
The JSON-merge fails in special cases when going from a long createOptions to a shorter one.

## Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
1. Create a deployment A with a module with a really long createOptions. It must be long enough so it's divided into `createOptions` and `createOptions01`.
2. Install it on a device.
3. Create a deployment B with the same module as in A but make the `createOptions` shorter so it only get a `createOptions` and not a `createOptions01`.
4. Change the deployment from A to B on the device.
5. edgeAgent will halt the update and complain about broken JSON data.

## Context (Environment)

### Output of `iotedge check`

Click here

```

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
‼ aziot-identity-service package is up-to-date - Warning
Installed aziot-identity-service package has version 1.4.3 but 1.4.9 is the latest stable version available.
Please see https://aka.ms/aziot-update-runtime for update instructions.
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
‼ aziot-edge package is up-to-date - Warning
Installed IoT Edge daemon has version 1.4.9 but 1.4.41 is the latest stable version available.
Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
33 check(s) succeeded.
2 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

```

### Device Information
* Host OS [e.g. Ubuntu 22.04, Windows Server IoT 2019]: Custom Yocto dist
* Architecture [e.g. amd64, arm32, arm64]: arm64
* Container OS [e.g. Linux containers, Windows containers]: Linux containers

### Runtime Versions
* aziot-edged [run `iotedge version`]: 1.4.9
* Edge Agent [image tag (e.g. 1.0.0)]: 1.5.31
* Edge Hub [image tag (e.g. 1.0.0)]: 1.5.31
* Docker/Moby [run `docker version`]: 20.10.25-ce

## Logs

aziot-edged logs

```

```

edge-agent logs

```

<4> 2025-12-04 15:23:40.269 +00:00 [WRN] - Reconcile failed because of invalid configuration format
Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.ConfigFormatException: Agent configuration format is invalid.
---> Newtonsoft.Json.JsonReaderException: Error parsing comment. Expected: *, got d. Path '', line 1, position 416.
at Newtonsoft.Json.JsonTextReader.ParseComment(Boolean setToken)
at Newtonsoft.Json.JsonTextReader.Read()
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig.GetCreateOptions(String createOptions) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 82
at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig..ctor(String image, String createOptions, Option`1 digest) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 28
at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig.DockerConfigJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 213
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type)
at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.TypeSpecificJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 104
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.Deserialize[TU](String json) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 63
at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.Deserialize(String json) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 56
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.UpdateDeploymentConfig(TwinCollection desiredProperties) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 435
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.UpdateDeploymentConfig(TwinCollection desiredProperties) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 447
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.ApplyPatchAsync(TwinCollection desiredProperties, TwinCollection patch) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 411
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132

```

edge-hub logs

```

```

## Additional Information
More detailed example.
Deployment A has a module that looks like this. Note that there is a `createOptions` and a `createOptions01`.
```
"mosquitto": {
"restartPolicy": "always",
"settings": {
"createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"Privileged\":true,\"ExtraHosts\":[\"host.docker.internal:host-gateway\"],\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}],\"9001/tcp\":[{\"HostPort\":\"9001\"}],\"1888/tcp\":[{\"HostPort\":\"1888\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\",\"",
"image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0",
"createOptions01": "/dev:/dev\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}"
},
"startupOrder": 100,
"status": "running",
"type": "docker"
},
```

Deployment B has a module that looks like this. Not only a `createOptions`.
```
"mosquitto": {
"restartPolicy": "always",
"settings": {
"createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}",
"image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0"
},
"startupOrder": 100,
"status": "running",
"type": "docker"
},
```

In `EdgeAgentConnections.cs` (ApplyPathAsync) a JSON merge between A and B i done.

```
string mergedJson = JsonEx.Merge(desiredProperties, patch, true);
```

Sine the class JsonEx doesn't know that `createOptions` and `createOptions01`are special (chunked) it will merge it into this:

```
"mosquitto": {
"restartPolicy": "always",
"settings": {
"createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}",
"image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0",
"createOptions01": "/dev:/dev\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}"
},
"startupOrder": 100,
"status": "running",
"type": "docker"
},
```

The result is that `createOptions01` is still left in the JSON. When later DockerConfig.cs takes that JSON object to create a docker instance it will see `createOptions` and `createOptions01` and use them as a chunked string. The result in invalid JSON which leads to a halt in the deployment change.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.