microsoft / microsoft/amplifier-bundle-dot-runner
Resume ladder cannot recover a fuse-killed run even with a raised max_pipeline_duration
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 2
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 88
Description
## Context
Follow-up from review of `fix/fuse-node-granularity` (attractor-674, PR TBD).
That PR makes `max_pipeline_duration` enforce at node granularity: a node that
outlives the remaining fuse budget is now cancelled in-process instead of
depending on an external hard-kill. On mid-node fuse expiry,
`PipelineEngine._terminate_fuse_mid_node` writes an honest per-node
`status.json` for the interrupted node (never added to `completed_nodes`),
and `run()`/`resume()`'s existing unconditional `_mark_run_completed()` flips
the on-disk checkpoint's `run_state` to `completed` -- never left `in_flight`.
## The gap
Because `run_state` is unconditionally flipped to `completed`, the resume
ladder (`load_checkpoint_for_resume`) refuses a fuse-killed checkpoint with
`CheckpointAlreadyCompletedError` -- consistent with every other terminal
engine outcome, and deliberately so (see `test_fuse_killed_checkpoint_refused_cleanly_by_resume_ladder`).
That refusal is honest, but it forecloses what seems like the natural
operator move after a fuse kill: "the budget was too tight, raise
`max_pipeline_duration` and resume from the last completed node." Today
there is no way to do that -- the only option is to start the whole pipeline
over from scratch, discarding whatever nodes genuinely completed before the
fuse fired.
## Ask
Decide (and document in `specs/EXTENSIONS.md` §15, if adopted) whether a
fuse-killed run should be distinguishable from an ordinary `completed` run
(e.g. a distinct `run_state` such as `fuse_terminated`, or a checkpoint field
recording the termination cause) so that a resume path *could* be added later
that re-validates the graph's `max_pipeline_duration` against the checkpoint
and allows continuation from the last completed node when the operator has
raised the ceiling. If the answer after design discussion is "no, always
restart" that is a legitimate outcome too -- but it should be a stated
decision in `specs/EXTENSIONS.md` rather than an implicit side effect of
reusing `run_state=completed` for a different termination cause.
Not a regression and not blocking the linked PR -- filed as agreed scope
control during that review.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read specs/EXTENSIONS.md §15, PipelineEngine._terminate_fuse_mid_node, load_checkpoint_for_resume, and the run()/resume() paths. Review test_fuse_killed_checkpoint_refused_cleanly_by_resume_ladder, then document a decision on whether fuse termination is distinguishable and resumable, including the raised-budget behavior, or explicitly record that runs must restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100