awslabs / awslabs/aws-embedded-metrics-node

Logging in a Lambda is not necessarily reliable

Open
#180 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
263
Forks
39
PR merge metrics
No merged PRs in 30d

Description

We've just found out (and it's obvious in hindsight) that doing `console.log()` in a Lambda Function doesn't guarantee that the logs make their way into CloudWatch Logs. Depending on the volume of logging, how quickly the Lambda ends after doing `console.log()`, and any throttling performed by CloudWatch Logs, the logs may or may not make it to the log group.

This is reasonable and logical if you think about it: Lambda cannot hold onto logs forever (space is finite), so they have a buffer of a certain size and duration, and if that buffer is exceeded then... too bad.

In our case, we were seeing a loss of ~0.1% of logs. We initially determined this by adding a `sleep(1_000)` at the end of our Lambda, that seemed to fix it, and then we confirmed with a Lambda Principal. Obviously a fixed sleep is a poor solution, so we plan to move to `PutLogEvents` explicitly to make sure that the logs we plan to send make it to CWL, or at least we'll know if they don't.

Here are our questions to you:

* How do you recommend we use this library for that scenario? Can we get the line that the library would have `console.log()`ged out and send it ourselves?
* An acknowledgement of this fact and a recommendation for how to solve it would be nice to see in the `README`. The fact that Lambda logs (and thereby metrics) may not be stored is not commonly known, but if we want to design reliable systems then the libraries we use have to be aware of these facts, and guide users in the right direction. Some people may decide that 0.1% of metrics lost is okay, and others might decide that it's not. But they need to be aware that they're making a choice.

> If your question is: "where is this lack of guarantees documented?" (as was my question), the answer to that was: "we didn't document that there would be any guarantees, so therefore there aren't"

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the README and the library's documented logging behavior, then determine whether it exposes the log line before it is sent to CloudWatch Logs. Document Lambda logging limitations, the reliability trade-off, and the recommended approach for users who need delivery guarantees; the issue is done when those guidance and usage questions are answered clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, observability-sre
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.