aws / aws/aws-parallelcluster

OverSubscribe in ParallelCluster 3.8.0 does not work

Open
#6,082 5 comments 0 reactions 0 assignees View on GitHub
3.x Feature Request
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

When trying to use `OverSubscribe` parameter via `CustomSlurmSettings` at the `Queue` level, the `pcluster` command produces a validation error

```
"configurationValidationErrors": [
{
"level": "ERROR",
"type": "CustomSlurmSettingsValidator",
"message": "Using the following custom Slurm settings at Queue level is not allowed: OverSubscribe"
},
```

If setting the same at `ComputeResources` level, the validator is ok with it but evidently this then leads to a broken SLURM configuration given that the `OVERSUBSCRIBE` parameter is only allowed at `Queue`/ `Partition` level.

In the `pcluster` source code we are associating `OverSubscribe` with the `Queue` level but put it on the `DENY_LIST`
https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/validators/slurm_settings_validator.py#L50

If removing `oversubscribe` from the `DENY_LIST`, `pcluster` will create a valid SLURM configuration, e.g. using this partial YAML code

```
....
SlurmQueues:
- Name: interactive
ComputeResources:
- Name: interactive
InstanceType: t2.xlarge
MaxCount: 20
MinCount: 1
Efa:
Enabled: FALSE
CustomSlurmSettings:
OverSubscribe: FORCE
...
```

Contributor guide

Open the contributing guide

Research direction

Start in cli/src/pcluster/validators/slurm_settings_validator.py at the DENY_LIST entry referenced by the issue. Check how CustomSlurmSettings are validated at Queue and ComputeResources levels, then use the supplied SlurmQueues YAML example to verify that Queue-level OverSubscribe is accepted and produces valid Slurm configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, hpc
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.