gchq / gchq/sleeper

Re-deploying same instance ID fails due to existing resources

Open
#6,423 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

Deploying a Sleeper instance, then tearing it down sometimes leaves resources behind. When re-deploying an instance of the same ID, we would expect it to work. However, CDK will often fail due to pre-existing Log groups, S3 buckets or DynamoDB tables. Making this situation more frustrating is that the user will have often waited anywhere from 10-30 minutes for a build to finish and partial deployment _before_ getting this error. Having fixed the error (by manually deleting the log group from AWS console for example), the user tries again, waits another 20 minutes only for CDK to fail again due a different type of pre-existing resource!

As a user, I want this to "just work", or at the very least, tell me about problems _at the start_ of deployment, not the end. A lot of the behaviour is enforced by CDK, so is not directly under our control; however, two options are:
1) Create an "remove all traces" script that can be run before re-deploying which, given a putative instance ID, checks for S3 buckets, CF stacks, DynamoDB tables, CW log groups that feature the "sleeper-{instance ID}" and removes them all. *Beware* Prefixes! Given two unrelated deployed instances: test and test2. Assuming "test" is the one we want to remove, just matching against all log group names will match "test" against "sleeper-test2" and remove those too!
2) Perform a CDK dry run as part of deployment which will create locally a manifest of all resources it intends to create. This could be inspected for the above resource types, then either fail early telling the user what needs to be deleted, or even better goes and deletes them.

Related #436.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing the CDK deployment entry point and how it derives sleeper-{instance ID} resource names; compare exact-prefix matching for cleanup across S3, CloudFormation, DynamoDB, and CloudWatch, and define an early failure or cleanup outcome before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.