Azure / Azure/cyclecloud-slurm

The dynamique partition does not work with the TaskPlugins task/cgroup

Open
#388 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
84
Forks
56
Avg merge
3d 17h
Merged PRs (30d)
1

Description

Hi,
I tried to use the dynamique partition. I could add the different types of nodes but the jobs where dying quickly. The error was

```
slurmstepd: error: common_file_write_uints: write value '39981' to '/sys/fs/cgroup/cpuset/slurm/uid_20006/job_6457/step_batch/cgroup.procs' failed: No space left on device
slurmstepd: error: unable to add pids to '/sys/fs/cgroup/cpuset/slurm/uid_20006/job_6457/step_batch'
slurmstepd: error: task_g_set_affinity: File exists
slurmstepd: error: _exec_wait_child_wait_for_parent: failed: Interrupted system call
slurmstepd: error: job_manager: exiting abnormally: Slurmd could not execve job
```

When I modified the slurm config to only use task/affinity in the TaskPlugins, any jobs could run. The hpc and the htc partition do not have this problem.

This is how I was creating the node sets:

```
scontrol create nodename=ukdri-cluster2-dyn4-[1-10] Feature=dyn,Standard_F4s_V2 cpus=4 State=CLOUD RealMemory=7782
scontrol create nodename=ukdri-cluster2-dyn4-[1-3] Feature=dyn,StandardF48s_V2 cpus=48 State=CLOUD RealMemory=93388
```

Thanks,
Thibaut

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with Slurm TaskPlugins set to task/cgroup and the dynamic node sets shown in the scontrol commands. Compare the result with task/affinity and with the hpc and htc partitions; done means jobs run successfully in the dynamic partition without the reported cgroup errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, linux
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.