aws / aws/amazon-ecs-agent

Add retries for publishing metrics & health checks

Open
#4,105 1 comment 0 reactions 0 assignees View on GitHub
contributions welcome good first issue kind/enhancement
Dominant language
Go
Stars
2.2k
Forks
662
Avg merge
3d 22h
Merged PRs (30d)
24

Description

### Summary
This is a request to add retries in the case of the agent failing to publish metrics or health check messages to TACS.

### Description
I noticed in my logs that I see cases where the ecs agent is emitting the message "Error publishing metrics" to the logs. From looking at [the code](https://github.com/aws/amazon-ecs-agent/blob/bc3cb997857126c4fe1b5fc6f5a0b192d3ef1f3b/ecs-agent/tcs/client/client.go#L131-L147) it looks like the `tcsClientServer.publishMessages` is reading metrics & health metrics from a channel and then emitting an error if the metrics were unable to be published. This behavior will result in either metrics or health checks failed to be reported to TACS when there is an error sending a message to TACS. For example, this could occur when a WS connection is closed from the server, which results in the client initiating a new connection.

### Expected Behavior
I would expect some kind of retry mechanism which would attempt to send the metrics or health checks over the connection. I don't see any retry logic further down the stack either ie: [`ClientServerImpl.MakeRequest`](https://github.com/aws/amazon-ecs-agent/blob/master/ecs-agent/wsclient/client.go#L420).

### Observed Behavior
The following log line:

~~~
05:20:14.273 | {"level":"warn","time":"2024-03-02T05:20:14.032","msg":"Error publishing metrics","error":"websocket: close sent"}
~~~

### Environment Details
Running on AL2 with kernel 5.10

### Supporting Log Snippets
See above.

Contributor guide

Open the contributing guide

Research direction

Start in ecs-agent/tcs/client/client.go at tcsClientServer.publishMessages, then inspect ecs-agent/wsclient/client.go at ClientServerImpl.MakeRequest and the connection handling around them. Define how failed metrics and health-check messages should be retried across websocket reconnects, and verify that the messages are eventually delivered without causing unbounded retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.