apache / apache/flink-agents

[Feature] Cross-language (Java/Python) agent spec parity test harness

Open
#1,064 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.

### Description

The project deliberately keeps Java and Python semantically aligned (AGENTS.md: "Public API changes must keep Java, Python, and YAML APIs semantically aligned"), but drift between the two runtimes keeps surfacing as separately-discovered bugs and tech-debt items rather than being caught systematically: #936 (non-happy-path response handling diverges between Java and Python for OpenAI-family connections), #1055/#1056 (umbrella reviews of API consistency for 0.4), #1016 (align Java and Python recovery model), and #687 (Python-side EventListener still missing to match the Java SDK). Each of these was found by manual inspection or by a user hitting the gap in production, not by a check that runs on every change.

What value does it bring: a shared, versioned test harness that defines the *same* small set of representative agent behaviors — a tool call, a structured-output request, an error path, a recovery-after-restart scenario — as data-driven fixtures, then runs each fixture against both the Java and the Python runtime and diffs the observable outcome (emitted events, final ChatMessage/tool outcome shape, error handling). A mismatch fails CI with a concrete diff instead of relying on someone noticing behavioral drift after the fact.

This complements, rather than duplicates, the ongoing manual audits (#1055/#1056/#1016) — it's the regression-prevention layer underneath them, so a parity issue those audits already fixed doesn't quietly reopen in a later change.

Do you already have a solution in mind: no PR yet — this is a proposal to check the direction is one the project wants before committing time to a specific harness shape (fixture format, where it lives — `e2e-test/` vs. its own module, how much of the outcome-shape comparison can be automated vs. needs an explicit per-fixture assertion).

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading AGENTS.md and reviewing the existing e2e-test/ location alongside the related parity issues. Define the fixture format and comparison scope, then confirm that representative behaviors run against both runtimes and that CI reports a concrete diff when outcomes diverge.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.