hashicorp / hashicorp/nomad

Add jitter or spread to periodic job scheduling

Open
#17,024 3 comments 1 reaction 0 assignees View on GitHub
theme/batch theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal

Given a job spec such as:

```hcl
job "ansible-cron" {
type = "sysbatch"

periodic {
cron = "@hourly"
}
}
```

There's currently no way to, as far as I can tell, to spread the schedule across over a period of time.

It's generally in a lot of cases that you want something to run hourly, or daily, but not every host in your cluster at the same exact time, you just want them to happen periodically.

### Use-cases

In our case, one of our use-cases of nomad is to manage running ansible on the underlying host periodically, but would like to not fleet-wide run ansible on every host at exactly the same time. Being able to spread the schedule across all hosts over a period of time would be ideal, and help mitigate any potential widespread issues. If for example, an ansible run is bad, we can interrupt it before it takes out the entire fleet rather than them all getting blasted at the exact same time.

It's also pretty common to just add jitter or spread to periodic jobs for resource contention in shared resources. Apt repos, etc. Everyone is pretty keenly aware of the top-of-the-hour spikes from everyone's cronjobs across the globe running at the exact same time.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name any files, tests, or entry points. Start by locating Nomad's periodic job scheduling implementation and its existing tests, then determine how schedule spreading would be represented in the job specification and validated across hosts. Done means periodic jobs can apply a bounded spread without synchronizing every host.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.