Improve CloudWatch alarms
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper, I want all CloudWatch alarms in an instance to have clear actions, so that anything that can be done automatically will be done, and I can see what I need to do when an alarm occurs.
### Description / Background
An instance of Sleeper creates a number of CloudWatch alarms for when SQS messages reach a dead letter queue. These could be useful, but currently by default these alarms don't trigger any actions. There's an instance property to set an email address to notify.
Some tools that scan AWS configuration flag it up as a problem when there's a CloudWatch alarm with no configured actions. There may be things we can do by default without needing an email address.
We'd like to review our current CloudWatch alarms and add any potential automation.
### Technical Notes / Implementation Details
We can raise separate issue for any potential changes.
We can look at every alarm and see what should be done.
Currently when a compaction job ends up on a dead letter queue, this raises an alarm, but leaves the job's input files assigned to the job. That means those files will never be compacted unless the job is run successfully. We could add automation to unassign the files from the job.
We have an issue to alarm when any ECS tasks fail to start:
- https://github.com/gchq/sleeper/issues/6099
This is also related:
- https://github.com/gchq/sleeper/issues/5545
Contributor guide
Assessment
This issue has not been assessed yet.