barelyhuman / barelyhuman/mac-runners-manager

Draining can kill running jobs

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

When draining, there's a good chance the runner picks up a job and then the drain kills it.

This race is because the job pick and the job state on Github might have a delay, it might be more reliable to have the SSH command verify if the runner has picked something up.

Since these are ephemeral, a `job picked` or another similar text can be used to find out if a job was picked, in which case the drain call needs to be re-checked by the manager.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the drain call through the manager and the SSH command handling. Check how the runner reports that a job was picked up and how the manager decides draining is complete. Done means a job picked during the race is detected and the drain operation is checked again instead of killing the running job.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.