[Feature] Dry-run / replay mode for validating agent behavior against production-shaped state before deploying
- Dominant language
- Java
- Stars
- 452
- Forks
- 167
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 49
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
What problem does it solve: today, validating a behavior change to an agent (a new prompt, a changed tool set, an updated routing strategy) against realistic state generally means either testing against synthetic fixtures or deploying the change and watching production. Flink Agents already carries durable per-key state through Flink's own checkpoint/savepoint mechanism (the recovery model #1016 is about aligning), which is exactly the ingredient a safer middle option needs.
What value does it bring: a dry-run mode that takes a savepoint from a running (or previously run) agent job, replays a defined set of keys/events against a *candidate* agent definition (the new prompt, tool set, or routing strategy under evaluation), and reports the resulting decisions/outputs side-by-side with what the current definition would produce for the same input — without those replayed actions taking effect against any real external tool, model call side effect that isn't idempotent, or downstream sink. It's the same idea as a Flink savepoint-based canary, specialized to "did my agent's behavior actually change the way I expect, on data shaped like production."
Do you already have a solution in mind: no PR yet — the open design question is how much of the replay path can reuse existing durable-state and event-recording machinery (the same Event Log format #970's Phase 1 OTel work already consumes) versus needing a genuinely new replay-only execution mode, so I'd like input on which direction the maintainers would actually want to carry before proposing a concrete shape.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reading the recovery-model discussion in #1016 and the Event Log format described in #970's Phase 1 OTel work. Compare how much existing durable-state and event-recording machinery could support replay against where a replay-only execution mode may be needed. Done is not yet defined: the issue is seeking maintainer input on the design before a concrete implementation is proposed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai-infra-agents
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100