gchq / gchq/sleeper

Monitor and act on frozen compactions

Open
#7,119 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs-refinement under-discussion
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

Based off #7118 , the compaction Job code should asynchronously monitor a running compaction in a container to detect if it has frozen. This could be by polling the compaction for the current number of rows processed and if no change is seen in a configurable time window, assume the compaction is frozen.

Remedial action probably wants some further discussion, but we could do some combination of:
1) Send an administrative alert, produce AWS Metric for display on dashboard
2) Terminate the compaction container
3) Put compaction job back on SQS queue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.