iterative / iterative/cml.dev

guide: GH resuming workflow

Open
#207 4 comments 1 reaction 0 assignees View on GitHub
documentation epic p1-important
Dominant language
TypeScript
Stars
13
Forks
22
PR merge metrics
No merged PRs in 30d

Description

Add a self-hosted long-running example to https://cml.dev/doc/cml-with-dvc (or somewhere else)

1. GH action launches "self-hosted" GCP/AWS using `cml runner --reuse --labels=cml` and probably `--cloud-spot`
2. GH action runs the rest of the workflow on the "self-hosted" runner using `runs-on: [self-hosted, cml]` and `timeout-minutes: 50400`
4. If GH action is about to timeout, CML will restart the workflow

- i.e. https://cml.dev/doc/self-hosted-runners?tab=GitHub#allocating-cloud-compute-resources-with-cml
- The key is requesting GH's maximum `timeout-minutes: 50400` - this signals to CML to restart the workflow just before timeout.
- write code to cache results so that the restarted workflow will use previous results (e.g. use https://dvc.org/doc/user-guide/experiment-management/checkpoints#caching-checkpoints and https://github.com/iterative/dvc/issues/6823)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.