microsoft / microsoft/agent-framework
[Feature] Workaround for tool call side-effect replay on checkpoint-based retry after executor failure
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
## Description
When an `AgentExecutor` fails mid-superstep (e.g., the underlying agent/LLM call crashes after some tool calls have already executed), the workflow engine does not save a checkpoint for the failed superstep. On resume from the last successful checkpoint, the entire agent turn is replayed from scratch — including tool calls that already completed and produced real-world side effects.
**What problem does it solve?**
Given a workflow `AgentExecutor "A" → AgentExecutor "B"`:
1. Superstep 1: A runs successfully → checkpoint saved
2. Superstep 2: B receives A's output, starts `agent.run()`:
- Tool call `send_email(to=bob)` → executed, email sent
- Tool call `update_database(status=done)` → executed, DB updated
- Tool call `notify_slack(channel=ops)` → crashes (e.g., network timeout)
3. Exception propagates before checkpoint creation (`_runner.py` line 145), no checkpoint saved for superstep 2
4. Resume from checkpoint 1: B re-runs the entire agent turn
- `send_email(to=bob)` → **duplicate email sent**
- `update_database(status=done)` → **duplicate DB write**
This happens because:
- Checkpoints are created at superstep boundaries, **after** successful completion (`_runner.py` line 145)
- If an executor fails, the exception propagates at `_runner.py` lines 127-131, before reaching checkpoint creation
- `AgentExecutor` treats the entire `agent.run()` as atomic — there is no per-tool-call checkpointing
- The executor has no awareness it is being retried vs. running for the first time
**Language/SDK**: Python
**Question**
This might be intentional given the Pregel superstep model, but is there a recommended workaround for users whose tools have external side effects? For example, is there a way to make the executor or tools aware that a retry is happening so they can skip already-completed operations?
Contributor guide
Assessment
This issue has not been assessed yet.