apache / apache/flink-agents

[Feature] Dry-run / replay mode for validating agent behavior against production-shaped state before deploying

Open
#1,066 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.

### Description

What problem does it solve: today, validating a behavior change to an agent (a new prompt, a changed tool set, an updated routing strategy) against realistic state generally means either testing against synthetic fixtures or deploying the change and watching production. Flink Agents already carries durable per-key state through Flink's own checkpoint/savepoint mechanism (the recovery model #1016 is about aligning), which is exactly the ingredient a safer middle option needs.

What value does it bring: a dry-run mode that takes a savepoint from a running (or previously run) agent job, replays a defined set of keys/events against a *candidate* agent definition (the new prompt, tool set, or routing strategy under evaluation), and reports the resulting decisions/outputs side-by-side with what the current definition would produce for the same input — without those replayed actions taking effect against any real external tool, model call side effect that isn't idempotent, or downstream sink. It's the same idea as a Flink savepoint-based canary, specialized to "did my agent's behavior actually change the way I expect, on data shaped like production."

Do you already have a solution in mind: no PR yet — the open design question is how much of the replay path can reuse existing durable-state and event-recording machinery (the same Event Log format #970's Phase 1 OTel work already consumes) versus needing a genuinely new replay-only execution mode, so I'd like input on which direction the maintainers would actually want to carry before proposing a concrete shape.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the recovery-model discussion in #1016 and the Event Log format described in #970's Phase 1 OTel work. Compare how much existing durable-state and event-recording machinery could support replay against where a replay-only execution mode may be needed. Done is not yet defined: the issue is seeking maintainer input on the design before a concrete implementation is proposed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ai-infra-agents
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.