hashicorp / hashicorp/nomad

[feature] Delayed job / Run once at a certain time

Open
#4,611 6 comments 7 reactions 0 assignees View on GitHub
theme/batch theme/jobspec type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

We recently ran into a scenario where we wanted to run a batch job exactly one time but at a certain time (by timestamp) in the future. This is likely analogous to a "delayed job" or "delayed message" construct in the queue and stream processing world.

In our case, our system is reacting to incoming events and launching dispatch jobs into the cluster as those events arrive. Under certain conditions where events are arriving for the same work "target", we want to throttle the jobs being dispatched and do something like "dispatch this job in X minutes".

We already use periodic and dispatch jobs heavily, but could not find a nomad native way to do this cleanly.

Possible solutions:

#### Use an external queue that implements the delayed message.

When the message is processed after the delay, submit a regular batch job or dispatch a parameterized job. An obvious solution and no changes are required to nomad, but adds an external dependency.

#### Implement the delay within the job itself

For this, a job would be dispatched a parameterized job with a "run_at" timestamp as part of the meta or payload. Nomad would allocate resources and start the job. The job's internal logic would sleep or wait until the specified time and then begin running.

#### Submit a periodic job with a cron specification that is constrained to one exact moment.

This would work, but the periodic job itself will exist in the cluster after the single run is triggered until manually deleted. This feels like abuse of the periodic job. However, if nomad could recognize that the cron expression will never trigger again and remove the job itself, this could work.

#### Allow a "run_at" or similar parameter for batch type jobs.

This seems to provide a straightforward nomad native solution to both the "delayed job" and "run once at a certain time" scenarios. No cluster resources are allocated while waiting, the workflow has the standard durability of any nomad job, and no additional external dependencies or application changes are required.

This could be implemented as a [dispatch option](https://www.nomadproject.io/docs/commands/job/dispatch.html#dispatch-options) (i.e. --run-at "2018-08-23 20:00:00") for parameterized jobs and as a [run option](https://www.nomadproject.io/docs/commands/job/run.html#run-options) for regular batch jobs.

If this were to be built, adding a sentence in the docs about not using this technique in place of a real queue for some workloads may be advisable.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked job dispatch and job run option documentation, then trace how batch and parameterized jobs are submitted. The issue proposes a durable run_at timestamp without allocating resources while waiting; the scope and implementation entry points would need to be established before work begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.