MetOffice / MetOffice/dagrunner

Will Cylc be added as a scheduler?

Open
#92 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Are there any plans to add Cylc as a scheduler for DAGRunner? Presumably this would be required for using it for the IMPROVER suite at some point. Or if not are there alternative cycling schedulers considered?

I can see from the [demo notebook](https://github.com/MetOffice/dagrunner/blob/main/docs/demo.ipynb) (section "Customised graph construction") that you can parameterise generating the networkx DiGraph object, e.g. with a cycle parameter. However, it's not clear to me how this could be used as a replacement/adjunct for a cycling scheduler. I can also see in the [IMPROVER suite design pack](https://github.com/MetOffice/improver_suite/wiki/Design-pack#technical-design) that there's a diagram that references cycling schedulers including Cylc and ECFLOW.

I'm interested in exploring the use of DAGRunner for a verification workflow. The aim here is to be able to run "offline" on an archive of IMPROVER outputs, e.g. from a hindcast, but also to be able to run in (delayed) realtime to verify outputs as observations become available. And in the long run, I suppose, we will transition our IMPROVER suite from paraflow to DAGRunner.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the demo notebook, especially the “Customised graph construction” section, and compare its cycle parameter with the scheduler role described in the IMPROVER suite design pack. Clarify whether Cylc integration or another cycling scheduler is wanted, including offline and delayed-realtime verification. Done requires an agreed scope and maintainer decision, since the issue currently asks an open-ended design question.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.