aws / aws/amazon-cloudwatch-agent

If only one EMF metric is sent before shutdown, metric is lost

Open
#768 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Describe the bug**

When a single EMF metric is sent, and the agent is immediately shutdown after that metric is sent, the metric is lost.

The logs look like:

```
2023-06-20T04:26:50Z I! [inputs.socket_listener] Listening on tcp://[::]:25888
--
2023-06-20T04:26:54Z I! Profiler is stopped during shutdown
2023-06-20T04:26:54Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-06-20T04:26:54Z I! [agent] Stopping running outputs
2023-06-20T04:26:54Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 180.286823ms before retrying.
2023-06-20T04:26:54Z E! [outputs.cloudwatchlogs] Stop requested after 0 retries to /ecs/NC-FeatureCalculator/EmbeddedMetrics/aca07567d7334356924db2cf7366ecac failed for PutLogEvents, request dropped.
```

The log stream /ecs/NC-FeatureCalculator/EmbeddedMetrics/aca07567d7334356924db2cf7366ecac is created.

What I believe is happening is that:

* [The agent attempts to put the log](https://github.com/aws/amazon-cloudwatch-agent/blob/2f5551345850d61a8b5d0d01ae9763234f851f15/plugins/outputs/cloudwatchlogs/pusher.go#L233)
* It fails because the log stream does not yet exist (this is the first/only event for my ECS task)
* [The agent creates the log stream](https://github.com/aws/amazon-cloudwatch-agent/blob/2f5551345850d61a8b5d0d01ae9763234f851f15/plugins/outputs/cloudwatchlogs/pusher.go#L274-L280)
* [The request is dropped without retrying because the agent is shutting down](https://github.com/aws/amazon-cloudwatch-agent/blob/2f5551345850d61a8b5d0d01ae9763234f851f15/plugins/outputs/cloudwatchlogs/pusher.go#L311-L313).

If I add a sleep statement after the metric scope is closed, I can see the retry occur and the metric sent to the created log stream group successfully, because it's processed before the shutdown occurs.

**Steps to reproduce**

Let me know if you need this, and I'll work it out.

**What did you expect to see?**

The log stream for my EMF metrics to contain an EMF metric, and that metric to be reflected in Cloudwatch logs.

**What did you see instead?**

The log stream for my EMF metrics exists but is empty, and the errors shown above.

**What version did you use?**

The current version from https://hub.docker.com/r/amazon/cloudwatch-agent, which appears to be 1.247359.1b252618.

**What config did you use?**

Config: {
"logs": {
"metrics_collected": {
"emf": {},
}
}
}

**Environment**

ECS sidecar

**Additional context**

NA

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.