[Feature Request] Allow shutdown/termination scripts in Slurm clusters
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
## Feature request description
Allow the possibility to run a script **before** the EC2 Instance Compute Node is terminated. A suitable location may be the [scaling section](https://docs.aws.amazon.com/parallelcluster/latest/ug/scaling-section.html), referencing an S3 location from where the script can be downloaded (similar approach to pre-init and post-init scripts).
## Background and details
- Starting from version [2.9.0](https://github.com/aws/aws-parallelcluster/blob/develop/CHANGELOG.md#290), scaling logic with Slurm scheduler was optimised, and is no longer based on Auto Scaling groups. Using the [Auto Scaling lifecycle hooks](https://aws.amazon.com/blogs/infrastructure-and-automation/run-code-before-terminating-an-ec2-auto-scaling-instance/) is no longer possible, for running custom actions when the nodes are terminated.
- There is no really a native way for implementing this. The Compute Nodes are managed from the Master node, which will issue the 'TerminateInstances' API Call for directly terminating the nodes.
## Why this can be relevant?
In large ecosystems and setups (which matches most of the Parallel Cluster use cases), there might a the need to run clean-up scripts, Active Directory de-registration, final reporting scripts, etc.
Contributor guide
Research direction
Start with the scaling section and the existing pre-init/post-init script approach described in the issue, then trace the Master node path that issues the EC2 TerminateInstances API call. Done should include an agreed configuration for a termination script sourced from S3 and execution before Compute Nodes are terminated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100