awslabs / awslabs/aws-embedded-metrics-python
awslogs driver for ECS - the newest log is not sent to CW if it is in EMF format
- Dominant language
- Python
- Stars
- 231
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
I'm seeing a weird behaviour of `awslogs` driver in ECS when it comes to sending logs to CloudWatch.
My web Python app (Docker, FastAPI) writes logs to the stdout using `aws-embedded-metrics-python`. This works just fine when running locally in Python or in Docker.
However, in AWS ECS, if the following two logs are sent:
```
1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...] blah
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format
```
only 1. is seen in CW.
If I swap the order to the following:
```
1. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
2. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}
```
then both 1. and 2. are seen in CW.
If I print something like this:
```
1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
3. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahbalah", "taskArn": "6900448c0b [...]}
```
Then 1., 2. and 3. are sent to CW.
It is as if `awslogs` ignored/buffered the last log if it happens to be in EMF format. Is this a documented / known behaviour or a bug?
Contributor guide
Assessment
This issue has not been assessed yet.