pytask-dev / pytask-dev/pytask
ENH: Preserve greedy execution with provisional tasks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 146
- Forks
- 14
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 34
Description
Is your feature request related to a problem?
When provisional task creation fails, a new DAG cannot be created and therefore execution cannot continue. The pytask session will stop.
Relevant paths include:
- provisional.py: failures while collecting generated tasks, running pytask_collect_modify_tasks, or validating task identities.
- provisional_utils.py: failures in recreate_dag, including dependency resolution, cycle detection, duplicate products, or scheduler rebuilding.
- recreate_dag also assigns session.dag before rebuilding the scheduler. If scheduler rebuilding fails, the session may contain a new DAG paired with an old scheduler.
Describe the solution you'd like
A better behavior would be to handle the whole process like a transaction and when any of the steps of registering these new provisional tasks fails, they are collected as an execution error and execution continues with the old DAG.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read provisional.py and provisional_utils.py to trace provisional task collection, identity validation, recreate_dag, and scheduler rebuilding. Define the transaction boundary and failure path so an error preserves the old DAG and scheduler while execution continues with that DAG.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100