I want some way to be able to define and track things to run on a schedule like cron jobs
- Dominant language
- Python
- Stars
- 60
- Forks
- 18
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 38
Description
I'd like to be able to execute `calkit run`, and e.g., if there's something that should be done every day, it is done if it hasn't already been done.
Imagine a DVC pipeline like:
```yaml
stages:
my-daily-job:
cmd: calkit run --procedure my-daily-job
deps:
- my-script.py
outs:
- results
always_changed: true # This will run every time
process-results:
cmd: python process.py
deps:
- process.py
- results
outs:
- processed
```
Maybe it's enough to make the `my-daily-job` stage smart enough to be very cheap to run if it already has been. Or we could define an automated procedure that saves its run to a log and has a way to not repeat itself. Maybe not that useful.
```yaml
procedures:
- name: my-daily-job
automated: true
no_repeat_frequency: 1d # Will only allow us to run it once per day
steps:
- python my-script.py
```
Also brings up the question of things that need to be done for each chunk of some collection, which could be days. Though maybe that's better handled with a DVC foreach stage? This is related to #38. We sort of want a way to define our parameter list or matrix externally rather than in the DVC pipeline itself.
With Calkit pipeline syntax, we could possibly do something like:
```yaml
pipeline:
stages:
my-daily-job:
kind: python-script
script_path: run.py
schedule: daily
```
Then we automatically output a file like `.calkit/scheduling/today.txt` that the auto-generated DVC stage depends on. This way, we don't need to write in idempotent logic into the script itself.
Contributor guide
Assessment
This issue has not been assessed yet.