baobabsoluciones / baobabsoluciones/cornflow
Split worklfows in DAG
- Dominant language
- Python
- Stars
- 21
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Currently the DAG of a model can perform the following workflows:
- check input data, solve, check solution data.
- check input and solution data alone.
With the changes currently in development a new task is added after the solving task to generate a report.
In the future these workflows should be separated into three different DAGs so that we can independently run the following wortkflows:
- given an execution id: run the input data checks, run the model, run the solution checks.
- given an execution id: generate the report
- given an instance id or an execution id: run the input data checks and the solution checks.
That would allow us to run this workflows separately and be able to have a better control over the concurrency of the different tasks.
The activate_dags script should create all the needed DAGs and from cornflow we should have different endpoints to control the different workflows.
In following comments we will add the list of tasks that are needed in order to develop the feature.
Contributor guide
Research direction
Start with the activate_dags script and the existing Cornflow workflow controls. Map the current input-check, solve, solution-check, and report tasks before deciding how the three independent DAGs and their endpoints should be exposed. Done means each workflow can be run separately with the stated execution or instance identifiers and concurrency can be controlled independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100