microsoft / microsoft/amplifier-bundle-dot-runner

Resume ladder cannot recover a fuse-killed run even with a raised max_pipeline_duration

Open
#35 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
2
Avg merge
4h 6m
Merged PRs (30d)
88

Description

## Context

Follow-up from review of `fix/fuse-node-granularity` (attractor-674, PR TBD).

That PR makes `max_pipeline_duration` enforce at node granularity: a node that
outlives the remaining fuse budget is now cancelled in-process instead of
depending on an external hard-kill. On mid-node fuse expiry,
`PipelineEngine._terminate_fuse_mid_node` writes an honest per-node
`status.json` for the interrupted node (never added to `completed_nodes`),
and `run()`/`resume()`'s existing unconditional `_mark_run_completed()` flips
the on-disk checkpoint's `run_state` to `completed` -- never left `in_flight`.

## The gap

Because `run_state` is unconditionally flipped to `completed`, the resume
ladder (`load_checkpoint_for_resume`) refuses a fuse-killed checkpoint with
`CheckpointAlreadyCompletedError` -- consistent with every other terminal
engine outcome, and deliberately so (see `test_fuse_killed_checkpoint_refused_cleanly_by_resume_ladder`).

That refusal is honest, but it forecloses what seems like the natural
operator move after a fuse kill: "the budget was too tight, raise
`max_pipeline_duration` and resume from the last completed node." Today
there is no way to do that -- the only option is to start the whole pipeline
over from scratch, discarding whatever nodes genuinely completed before the
fuse fired.

## Ask

Decide (and document in `specs/EXTENSIONS.md` §15, if adopted) whether a
fuse-killed run should be distinguishable from an ordinary `completed` run
(e.g. a distinct `run_state` such as `fuse_terminated`, or a checkpoint field
recording the termination cause) so that a resume path *could* be added later
that re-validates the graph's `max_pipeline_duration` against the checkpoint
and allows continuation from the last completed node when the operator has
raised the ceiling. If the answer after design discussion is "no, always
restart" that is a legitimate outcome too -- but it should be a stated
decision in `specs/EXTENSIONS.md` rather than an implicit side effect of
reusing `run_state=completed` for a different termination cause.

Not a regression and not blocking the linked PR -- filed as agreed scope
control during that review.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read specs/EXTENSIONS.md §15, PipelineEngine._terminate_fuse_mid_node, load_checkpoint_for_resume, and the run()/resume() paths. Review test_fuse_killed_checkpoint_refused_cleanly_by_resume_ladder, then document a decision on whether fuse termination is distinguishable and resumable, including the raised-budget behavior, or explicitly record that runs must restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.