Add retries for publishing metrics & health checks
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 662
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 24
Description
### Summary
This is a request to add retries in the case of the agent failing to publish metrics or health check messages to TACS.
### Description
I noticed in my logs that I see cases where the ecs agent is emitting the message "Error publishing metrics" to the logs. From looking at [the code](https://github.com/aws/amazon-ecs-agent/blob/bc3cb997857126c4fe1b5fc6f5a0b192d3ef1f3b/ecs-agent/tcs/client/client.go#L131-L147) it looks like the `tcsClientServer.publishMessages` is reading metrics & health metrics from a channel and then emitting an error if the metrics were unable to be published. This behavior will result in either metrics or health checks failed to be reported to TACS when there is an error sending a message to TACS. For example, this could occur when a WS connection is closed from the server, which results in the client initiating a new connection.
### Expected Behavior
I would expect some kind of retry mechanism which would attempt to send the metrics or health checks over the connection. I don't see any retry logic further down the stack either ie: [`ClientServerImpl.MakeRequest`](https://github.com/aws/amazon-ecs-agent/blob/master/ecs-agent/wsclient/client.go#L420).
### Observed Behavior
The following log line:
~~~
05:20:14.273 | {"level":"warn","time":"2024-03-02T05:20:14.032","msg":"Error publishing metrics","error":"websocket: close sent"}
~~~
### Environment Details
Running on AL2 with kernel 5.10
### Supporting Log Snippets
See above.
Contributor guide
Research direction
Start in ecs-agent/tcs/client/client.go at tcsClientServer.publishMessages, then inspect ecs-agent/wsclient/client.go at ClientServerImpl.MakeRequest and the connection handling around them. Define how failed metrics and health-check messages should be retried across websocket reconnects, and verify that the messages are eventually delivered without causing unbounded retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100