baobabsoluciones / baobabsoluciones/cornflow

Split worklfows in DAG

Open
#545 0 comments 0 reactions 0 assignees View on GitHub
app client dags enhancement major change server
Dominant language
Python
Stars
21
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Currently the DAG of a model can perform the following workflows:

- check input data, solve, check solution data.
- check input and solution data alone.

With the changes currently in development a new task is added after the solving task to generate a report.

In the future these workflows should be separated into three different DAGs so that we can independently run the following wortkflows:

- given an execution id: run the input data checks, run the model, run the solution checks.
- given an execution id: generate the report
- given an instance id or an execution id: run the input data checks and the solution checks.

That would allow us to run this workflows separately and be able to have a better control over the concurrency of the different tasks.

The activate_dags script should create all the needed DAGs and from cornflow we should have different endpoints to control the different workflows.

In following comments we will add the list of tasks that are needed in order to develop the feature.

Contributor guide

Open the contributing guide

Research direction

Start with the activate_dags script and the existing Cornflow workflow controls. Map the current input-check, solve, solution-check, and report tasks before deciding how the three independent DAGs and their endpoints should be exposed. Done means each workflow can be run separately with the stated execution or instance identifiers and concurrency can be controlled independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.