box / box/ClusterRunner

Build subjobs get reordered if slave is put into shutdown mode

Open
#226 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Currently when a build goes to send its next subjob to a slave and finds that the slave has been put into shutdown mode, the build correctly reinserts that subjob into its unstarted subjobs queue.

The issue is that the subjob goes to the end of the queue. It will now be the last subjob to get executed as opposed to the next one.

This is not a huge problem -- the build will still execute as normal -- but there is the possibility for a large hit to scheduling efficiency if this happens early in the build. (Since we always execute the longest subjobs first, postponing these first subjobs until the end of the build could increase the build time by 2x or more.)

This is a pretty rare case since we don't often put slaves into shutdown mode, but this should be a relatively easy fix. We could use something like `queue.PriorityQueue` keyed off of the subjob id instead of a standard FIFO queue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the shutdown handling path where a subjob is reinserted into the unstarted subjobs queue, then inspect how that queue orders work and how subjobs are dispatched to slaves. Preserve the intended longest-subjob-first ordering when a slave enters shutdown mode, and verify that the affected build still executes every subjob in the expected order.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.