aws-samples / aws-samples/1click-hpc

grafana monitoring not working for static resources

Open
#35 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
67
Forks
43
PR merge metrics
No merged PRs in 30d

Description

when we use a non-zero minimum in cluster config for resources, they get alive at cluster launch. then this job-related check will never have a value of True: https://github.com/aws-samples/1click-hpc/blob/7a833d4a56dd42d28836b91938168cd4ca841e28/modules/40.install.monitoring.compute.sh#L59

because this must be run in the root context, the only chance to do it is in the prolog script to attach it to a job, so basically the plan would be to

1. install the docker container anyway in post-install but do not start it
2. use prolog and epilog to start and stop the container depending on user's choice to monitor or not

the problem is how to send a signal about the job to prolog and epilog since the custom user env variables are not sent, and the job comment is not sent. Because per slurm manuals, we should not perform `scontrol` from prolog; this will impair the scaling of the jobs similarly to the API calls (this is related to #34 )

Looking at the variables available at prolog/epilog time I only have 2 ideas so far:

1. SLURM_PRIO_PROCESS Scheduling priority (nice value) at the time of submission. Available in SrunProlog, TaskProlog, SrunEpilog and TaskEpilog. We can `#SBATCH --nice 0` or some sensible value to uniquely identify the intention then use the TaskProlog and TaskEpilog to start/stop the monitoring container
2. use some crafted slurm job name like `[GM] my job name` then pick and interpret this from SLURM_JOB_NAME Name of the job. Available in PrologSlurmctld, SrunProlog, TaskProlog, EpilogSlurmctld, SrunEpilog and TaskEpilog. Meaning also the use of TaskProlog and TaskEpilog to start/stop the monitoring container

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.