awslabs / awslabs/aws-embedded-metrics-python

awslogs driver for ECS - the newest log is not sent to CW if it is in EMF format

Open
#109 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
231
Forks
44
PR merge metrics
No merged PRs in 30d

Description

I'm seeing a weird behaviour of `awslogs` driver in ECS when it comes to sending logs to CloudWatch.

My web Python app (Docker, FastAPI) writes logs to the stdout using `aws-embedded-metrics-python`. This works just fine when running locally in Python or in Docker.

However, in AWS ECS, if the following two logs are sent:
```
1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...] blah
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format
```
only 1. is seen in CW.

If I swap the order to the following:
```
1. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
2. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}
```
then both 1. and 2. are seen in CW.

If I print something like this:
```
1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
3. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahbalah", "taskArn": "6900448c0b [...]}
```
Then 1., 2. and 3. are sent to CW.

It is as if `awslogs` ignored/buffered the last log if it happens to be in EMF format. Is this a documented / known behaviour or a bug?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.