calkit / calkit/calkit

I want some way to be able to define and track things to run on a schedule like cron jobs

Open
#47 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
60
Forks
18
Avg merge
1d 16h
Merged PRs (30d)
38

Description

I'd like to be able to execute `calkit run`, and e.g., if there's something that should be done every day, it is done if it hasn't already been done.

Imagine a DVC pipeline like:

```yaml
stages:
my-daily-job:
cmd: calkit run --procedure my-daily-job
deps:
- my-script.py
outs:
- results
always_changed: true # This will run every time
process-results:
cmd: python process.py
deps:
- process.py
- results
outs:
- processed
```

Maybe it's enough to make the `my-daily-job` stage smart enough to be very cheap to run if it already has been. Or we could define an automated procedure that saves its run to a log and has a way to not repeat itself. Maybe not that useful.

```yaml
procedures:
- name: my-daily-job
automated: true
no_repeat_frequency: 1d # Will only allow us to run it once per day
steps:
- python my-script.py
```

Also brings up the question of things that need to be done for each chunk of some collection, which could be days. Though maybe that's better handled with a DVC foreach stage? This is related to #38. We sort of want a way to define our parameter list or matrix externally rather than in the DVC pipeline itself.

With Calkit pipeline syntax, we could possibly do something like:

```yaml
pipeline:
stages:
my-daily-job:
kind: python-script
script_path: run.py
schedule: daily
```

Then we automatically output a file like `.calkit/scheduling/today.txt` that the auto-generated DVC stage depends on. This way, we don't need to write in idempotent logic into the script itself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.