aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
(orchestration): Context quality metrics on task completion
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
## Component
API or orchestration
## Describe the feature
Emit and persist **context-quality signals** per task at completion: tool-call count, unique files touched, hydration token usage, retry/verify loops, turns consumed vs. budget, and optional “scope expansion” flag. Expose in API, CloudWatch metrics, and `bgagent stats`.
## Use case
High tool-call counts and repeated verify failures often indicate **missing harness context** rather than model weakness. Production teams use **tool calls per session** as a proxy: more curated skills/rules → fewer exploratory tool calls → faster task completion. Platform teams need this metric to prioritize repo onboarding and detect context drift after refactors.
## Proposed solution
1. Agent `progress_writer` / runner aggregates counters during execution.
2. Orchestrator writes `context_quality` object on task record and terminal `TaskEvent`.
3. CloudWatch embedded metric format for `ToolCallsPerTask`, `VerifyRetriesPerTask`.
4. Dashboard widget in operator dashboard; optional alarm on p95 tool calls per repo.
## Other information
- Complements roadmap **LLM-assisted trace analysis** (post-mortem) with cheap always-on counters.
- Pairs with **feature-service-catalog-context-hydration** and **Dynamic onboarding artifacts**.
## Acknowledgements
- [ ] I may be able to implement this feature
- [ ] This might be a breaking change
## Acceptance criteria
- [ ] Terminal tasks include `context_quality` in API responses when execution reached RUNNING.
- [ ] Metrics visible in CloudWatch with `repo` and `workflow_ref` dimensions.
- [ ] Documented interpretation guide in `EVALUATION.md`.
- [ ] Unit tests for aggregation logic in agent and handler layers.
Contributor guide
Research direction
Start by locating the agent progress_writer or runner, the orchestrator task record and terminal TaskEvent handling, and the operator dashboard and CloudWatch metric entry points. Trace how completion data reaches API responses, then review EVALUATION.md and existing handler and agent tests. Done means the listed acceptance criteria are covered across aggregation, persistence, metrics, dashboard visibility, documentation, and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend-api-design, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100