aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

(agent): Factory self-improvement audit automation

Open
#416 0 comments 0 reactions 0 assignees View on GitHub
agent-runtime enhancement
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

## Component

Workflows

## Describe the feature

Ship a reference **`coding/factory-audit-v1`** (or scheduled workflow) that periodically reviews recent task trajectories, prompt versions, and automation outcomes—proposing harness improvements (rules, standards, workflow YAML) as issues or draft PRs.

## Use case

Operators running mature background-agent fleets often want an **automation auditor** that reviews prompts and run logs to suggest harness improvements. ABCA has evaluation roadmap items but no packaged **closed-loop self-improvement** workflow operators can enable.

## Proposed solution

1. Scheduled trigger (roadmap) or manual `bgagent submit --workflow factory-audit-v1`.
2. Read-only access to `TaskEvents`, traces, merge outcomes for repo/time window.
3. Output: markdown report artifact in S3 + optional GitHub issue creation.
4. Human approves before any harness file auto-PR.

## Other information

- Complements **Evaluation pipeline**, **LLM-assisted trace analysis**, **A/B prompt experiments**.
- Repo-less or read-only repo mode acceptable.

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

## Acceptance criteria

- [ ] Reference workflow YAML and prompt shipped under `agent/workflows/`.
- [ ] Produces structured report with actionable recommendations.
- [ ] Does not auto-merge harness changes without HITL.
- [ ] Tests with fixture task history.

Contributor guide

Open the contributing guide

Research direction

Start with the existing workflows under agent/workflows/ and the manual bgagent submit --workflow factory-audit-v1 entry point, then inspect how TaskEvents, traces, and merge outcomes are exposed. Done means a reference workflow YAML and prompt produce a structured markdown report, cover fixture task history, and require HITL before any harness change is merged.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github, yaml
Domain
ai, devops, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.