pytask-dev / pytask-dev/pytask

ENH: Preserve greedy execution with provisional tasks

Open
#985 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
146
Forks
14
Avg merge
3d 16h
Merged PRs (30d)
34

Description

Is your feature request related to a problem?

When provisional task creation fails, a new DAG cannot be created and therefore execution cannot continue. The pytask session will stop.

Relevant paths include:

  • provisional.py: failures while collecting generated tasks, running pytask_collect_modify_tasks, or validating task identities.
  • provisional_utils.py: failures in recreate_dag, including dependency resolution, cycle detection, duplicate products, or scheduler rebuilding.
  • recreate_dag also assigns session.dag before rebuilding the scheduler. If scheduler rebuilding fails, the session may contain a new DAG paired with an old scheduler.
Describe the solution you'd like

A better behavior would be to handle the whole process like a transaction and when any of the steps of registering these new provisional tasks fails, they are collected as an execution error and execution continues with the old DAG.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read provisional.py and provisional_utils.py to trace provisional task collection, identity validation, recreate_dag, and scheduler rebuilding. Define the transaction boundary and failure path so an error preserves the old DAG and scheduler while execution continues with that DAG.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.