aws / aws/aws-parallelcluster

[Feature Request] Allow shutdown/termination scripts in Slurm clusters

Open
#2,491 4 comments 0 reactions 0 assignees View on GitHub
enhancement Feature Request waiting customer
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

## Feature request description

Allow the possibility to run a script **before** the EC2 Instance Compute Node is terminated. A suitable location may be the [scaling section](https://docs.aws.amazon.com/parallelcluster/latest/ug/scaling-section.html), referencing an S3 location from where the script can be downloaded (similar approach to pre-init and post-init scripts).

## Background and details
- Starting from version [2.9.0](https://github.com/aws/aws-parallelcluster/blob/develop/CHANGELOG.md#290), scaling logic with Slurm scheduler was optimised, and is no longer based on Auto Scaling groups. Using the [Auto Scaling lifecycle hooks](https://aws.amazon.com/blogs/infrastructure-and-automation/run-code-before-terminating-an-ec2-auto-scaling-instance/) is no longer possible, for running custom actions when the nodes are terminated.
- There is no really a native way for implementing this. The Compute Nodes are managed from the Master node, which will issue the 'TerminateInstances' API Call for directly terminating the nodes.

## Why this can be relevant?

In large ecosystems and setups (which matches most of the Parallel Cluster use cases), there might a the need to run clean-up scripts, Active Directory de-registration, final reporting scripts, etc.

Contributor guide

Open the contributing guide

Research direction

Start with the scaling section and the existing pre-init/post-init script approach described in the issue, then trace the Master node path that issues the EC2 TerminateInstances API call. Done should include an agreed configuration for a termination script sourced from S3 and execution before Compute Nodes are terminated.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.