Feature or Documentation Request - Continuous Deployment (eg. Blue/Green )

One major challenge I have had is the effort and roadblocks involved in releasing configuration changes and other improvements to active clusters. Oftentimes, it is impossible to make any required changes to a cluster because the compute fleet cannot be stopped or drained due to a running job, that often run for days or weeks at a time.

Many cluster update operations' [Update Policy](https://docs.aws.amazon.com/parallelcluster/latest/ug/using-pcluster-update-cluster-v3.html#pcluster-update-policy-definitions) either requires labour intensive activities or is flat-out not allowed, the implicit instruction being to just build a new cluster. Ideally ParallelCluster would support frequent incremental changes, for example changing [Tags](https://docs.aws.amazon.com/parallelcluster/latest/ug/Tags-v3.html)

Some other examples of challenges with making changes.
1. #6357 updating the SharedStorage cluster configuration detaches and recreates all of the shared storage volumes specified in the updated configuration
2. #6339 In-place upgrade causes rollback
3. #6214 accounting database migration fails

Not covered would be the operational updates made outside of the Parallel Cluster configuration files, such as [CustomActions](https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#Scheduling-v3-SlurmQueues-CustomActions) scripts, and [CustomSlurmSettings Include Files](https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#yaml-Scheduling-SlurmSettings-CustomSlurmSettings)

While all of these could be addressed individually, a sledgehammer approach that should cover all contingencies, even those we've not yet encountered, would be explicit support or otherwise guidance on [Blue/green deployments](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/bluegreen-deployments.html#:~:text=A%20blue%2Fgreen%20deployment%20is,running%20the%20new%20application%20version.).

The benefits would be:
1. ability to release much more frequent incremental improvements
2. greater support for a fully automated Continuous Deployment pipeline
3. ability to run automated smoke tests and integration tests such as submitting slurm test cases to validate the stack end-to-end
4. ability to run other manual UAT activities if required
5. ensure that the updated configuration works correctly before the users encounter errors
6. Non-disuptively support _all_ create and update operations, not just those explicitly supported by `pcluster update-cluster`
7. empowered to make arbitrary configuration changes without requiring manual actions such as stopping or draining compute fleets
8. Release / support a new version of your HPC software (eg. we link /ansys_inc/latest to /ansys_inc/v245 or whatever) and be confident the entire stack is working

Different use cases will obviously have a different mix of supporting resources such as users, persistent and transient data. For example, we keep our CFD application binaries and support scripts on an EFS filesystem that would need to be defined outside the individual cluster's SharedStorage configuration.

Our model is a "project" at a top level, with resources including
- a persistent data store (EFS binaries)
- s3 buckets for case, result and post-processed data
- IAM policies that are references by our IAM Identity Centre for our users
- IAM policies for attachment to EC2 resources for all cluster resources in the project
- additional security groups for attachment to EC2 resources to allow access to/from project resources liek s3 buckets

One level down from the project is the ParallelCluster concept of cluster, with the familiar which includes
head node, compute queues, login nodes, lustre filesystems for in-flight data with cluster and case-specific DRA prefixes out to the "project" s3 buckets

Blue/green deployments for clusters within a project would be a powerful capability

I have also opened a case with Enterprise Support #172291072100573

The issue names no repository files, tests, or entry points. Start by reviewing the pcluster update-cluster and deployment guidance linked in the request, then examine issues #6357, #6339, and #6214; done requires a defined, supported blue/green deployment capability or explicit guidance covering the described cluster-update scenarios.

Written by the indexing model from the issue text.