celerity / celerity/slurmactiond

Periodically check if a runner is stuck

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
13
Forks
4
PR merge metrics
No merged PRs in 30d

Description

I'm currently observing runners that get stuck after this output:

```
[INFO ] SLURM #9281: Running on gpuc2 in `/software-local/cirunner/slurmactiond/nvidia/3`
[INFO ] Runner 20 (slurm-nvidia-9281) connected for target nvidia
[DEBUG] SLURM #9281: Re-using existing Github Actions Runner installation
[INFO ] SLURM #9281: Generating runner registration token for celerity
[DEBUG] SLURM #9281: Sending GitHub API request POST https://api.github.com/orgs/celerity/actions/runners/registration-token
[INFO ] SLURM #9281: Unregistering previous runner if present
[DEBUG] SLURM #9281: config.sh:
[DEBUG] SLURM #9281: config.sh: # Runner removal
[DEBUG] SLURM #9281: config.sh:
[DEBUG] SLURM #9281: config.sh: Cannot connect to server, because config files are missing. Skipping removing runner from the server.
[DEBUG] SLURM #9281: config.sh: Does not exist. Skipping Removing .credentials
[DEBUG] SLURM #9281: config.sh: Does not exist. Skipping Removing .runner
[DEBUG] SLURM #9281: config.sh:
[INFO ] SLURM #9281: Registering runner slurm-nvidia-9281
[DEBUG] SLURM #9281: config.sh:
[DEBUG] SLURM #9281: config.sh: --------------------------------------------------------------------------------
[DEBUG] SLURM #9281: config.sh: | ____ _ _ _ _ _ _ _ _ |
[DEBUG] SLURM #9281: config.sh: | / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ |
[DEBUG] SLURM #9281: config.sh: | | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| |
[DEBUG] SLURM #9281: config.sh: | | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ |
[DEBUG] SLURM #9281: config.sh: | \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ |
[DEBUG] SLURM #9281: config.sh: | |
[DEBUG] SLURM #9281: config.sh: | Self-hosted runner registration |
[DEBUG] SLURM #9281: config.sh: | |
[DEBUG] SLURM #9281: config.sh: --------------------------------------------------------------------------------
[DEBUG] SLURM #9281: config.sh:
[DEBUG] SLURM #9281: config.sh: # Authentication
[DEBUG] SLURM #9281: config.sh:
```

This could be detected in slurmactiond through a timeout between `runner_connected` and `runner_listening`.

We should consider the failure mode of a runner getting stuck during job execution as well.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing where runner_connected and runner_listening are handled in slurmactiond. Define how a timeout should detect a runner stuck during registration, then examine how a runner stuck during job execution could be detected. Done should include handling both failure modes with observable behavior, though the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.