gchq / gchq/sleeper

Improve CloudWatch alarms

Open
#6,977 0 comments 0 reactions 0 assignees View on GitHub
enhancement epic
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want all CloudWatch alarms in an instance to have clear actions, so that anything that can be done automatically will be done, and I can see what I need to do when an alarm occurs.

### Description / Background

An instance of Sleeper creates a number of CloudWatch alarms for when SQS messages reach a dead letter queue. These could be useful, but currently by default these alarms don't trigger any actions. There's an instance property to set an email address to notify.

Some tools that scan AWS configuration flag it up as a problem when there's a CloudWatch alarm with no configured actions. There may be things we can do by default without needing an email address.

We'd like to review our current CloudWatch alarms and add any potential automation.

### Technical Notes / Implementation Details

We can raise separate issue for any potential changes.

We can look at every alarm and see what should be done.

Currently when a compaction job ends up on a dead letter queue, this raises an alarm, but leaves the job's input files assigned to the job. That means those files will never be compacted unless the job is run successfully. We could add automation to unassign the files from the job.

We have an issue to alarm when any ECS tasks fail to start:
- https://github.com/gchq/sleeper/issues/6099

This is also related:
- https://github.com/gchq/sleeper/issues/5545

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.