aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

Tooling: Conformance fixtures — golden PEV trajectories per workflow

Open
#508 0 comments 0 reactions 0 assignees View on GitHub
enhancement tooling
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

> **Roadmap:** Exposed project specifications; #236 eval harness
> **Priority:** P3

## Component

Other

## Describe the feature

Add `contracts/harness-trajectories/` with **golden fixtures** defining expected orchestrator/agent event sequences and terminal states for each shipped coding workflow. Enables harness-level conformance testing separate from model quality.

## Use case

- **ABCA Spec** claims conforming implementations pass golden trajectories.
- **CI** catches orchestrator/workflow regressions without live Bedrock.
- **Fork authors** validate alternative orchestrators against the same fixtures.

## Proposed solution

1. Directory layout:

```
contracts/harness-trajectories/
coding-new-task-v1.happy-path.json
coding-pr-iteration-v1.happy-path.json
coding-pr-review-v1.happy-path.json
README.md
```

2. Fixture schema: ordered `TaskEvent` types, required fields, terminal `status`, optional mocked agent responses.

3. Jest test in `cdk/test/contracts/` loads fixtures and validates against orchestrator simulation or recorded snapshots (no bundling live agent).

4. Link from harness glossary / ABCA Spec extension catalog.

### Acceptance criteria

- [ ] At least one golden trajectory per shipped coding workflow
- [ ] CI job fails on fixture regression
- [ ] README documents fixture schema and how to add scenarios (failure paths optional follow-up)
- [ ] No live AWS/Bedrock required in unit tests

## Other information

- **Enables:** Harness mutation regression suite issue
- **Related:** #236 eval harness, unified task replay API
- **Paper:** arXiv:2605.18747 replayability and harness-level eval

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the existing contract tests under cdk/test/contracts/ and the shipped coding workflows, then review the proposed contracts/harness-trajectories/ layout and related harness glossary or ABCA Spec links. Done means each shipped workflow has a documented golden trajectory, the README explains the schema and adding scenarios, CI detects regressions, and unit tests require no live AWS or Bedrock.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.