AWS X-Ray Daemon occasionally fails to send telemetry record to the service using the [`PutTelemetryRecord`](https://docs.aws.amazon.com/xray/latest/api/API_PutTelemetryRecords.html) API. The debug logs would show the error message like this:
```
2020-05-10T11:53:00Z [Info] Successfully sent batch of 1 segments (2.943 seconds)
2020-05-10T11:53:47Z [Debug] Send 1 telemetry record(s)
2020-05-10T11:54:49Z [Debug] Send 1 telemetry record(s)
2020-05-10T11:55:51Z [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed to unmarshal response error
status code: 400, request id:
caused by: UnmarshalError: failed decoding error message
00000000 3c 68 74 6d 6c 3e 0d 0a 3c 68 65 61 64 3e 3c 74 |..400 Bad Req|
00000020 75 65 73 74 3c 2f 74 69 74 6c 65 3e 3c 2f 68 65 |uest....

400 Bad|
00000060 20 52 65 71 75 65 73 74 3c 2f 68 31 3e 3c 2f 63 | Request

...|
00000080 0a 3c 2f 68 74 6d 6c 3e 0d 0a |...|

caused by: invalid character '<' looking for beginning of value
```

What appears to happen is that the daemon may be sending some wrong or missing parameters for the PutTelemetryRecord API generating the 400 error and then is unable to process the response which is an html instead of json.

This causes a lot of noise in the debug logs and needs to be handled by the daemon.
I did some digging and found out that when a telemetry record fails to send, it is re-queued with the next batch and sometimes the next batch would be sent out successfully. This is confusing since the 400 status code in the first place indicated that the telemetry record may be incorrect. I tried printing some of the telemetry records and found that they were fine as the only requird field for the [`TelemetryRecord`](https://docs.aws.amazon.com/xray/latest/api/API_TelemetryRecord.html) is the timestamp.

I will need to talk to the service team to find the root cause of the issue.