Azure / Azure/cyclecloud-slurm
The dynamique partition does not work with the TaskPlugins task/cgroup
- Dominant language
- Python
- Stars
- 84
- Forks
- 56
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 1
Description
Hi,
I tried to use the dynamique partition. I could add the different types of nodes but the jobs where dying quickly. The error was
```
slurmstepd: error: common_file_write_uints: write value '39981' to '/sys/fs/cgroup/cpuset/slurm/uid_20006/job_6457/step_batch/cgroup.procs' failed: No space left on device
slurmstepd: error: unable to add pids to '/sys/fs/cgroup/cpuset/slurm/uid_20006/job_6457/step_batch'
slurmstepd: error: task_g_set_affinity: File exists
slurmstepd: error: _exec_wait_child_wait_for_parent: failed: Interrupted system call
slurmstepd: error: job_manager: exiting abnormally: Slurmd could not execve job
```
When I modified the slurm config to only use task/affinity in the TaskPlugins, any jobs could run. The hpc and the htc partition do not have this problem.
This is how I was creating the node sets:
```
scontrol create nodename=ukdri-cluster2-dyn4-[1-10] Feature=dyn,Standard_F4s_V2 cpus=4 State=CLOUD RealMemory=7782
scontrol create nodename=ukdri-cluster2-dyn4-[1-3] Feature=dyn,StandardF48s_V2 cpus=48 State=CLOUD RealMemory=93388
```
Thanks,
Thibaut
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with Slurm TaskPlugins set to task/cgroup and the dynamic node sets shown in the scontrol commands. Compare the result with task/affinity and with the hpc and htc partitions; done means jobs run successfully in the dynamic partition without the reported cgroup errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, linux
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100