aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

(orchestration): Context quality metrics on task completion

Open
#415 0 comments 0 reactions 0 assignees View on GitHub
enhancement observability orchestration
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

## Component

API or orchestration

## Describe the feature

Emit and persist **context-quality signals** per task at completion: tool-call count, unique files touched, hydration token usage, retry/verify loops, turns consumed vs. budget, and optional “scope expansion” flag. Expose in API, CloudWatch metrics, and `bgagent stats`.

## Use case

High tool-call counts and repeated verify failures often indicate **missing harness context** rather than model weakness. Production teams use **tool calls per session** as a proxy: more curated skills/rules → fewer exploratory tool calls → faster task completion. Platform teams need this metric to prioritize repo onboarding and detect context drift after refactors.

## Proposed solution

1. Agent `progress_writer` / runner aggregates counters during execution.
2. Orchestrator writes `context_quality` object on task record and terminal `TaskEvent`.
3. CloudWatch embedded metric format for `ToolCallsPerTask`, `VerifyRetriesPerTask`.
4. Dashboard widget in operator dashboard; optional alarm on p95 tool calls per repo.

## Other information

- Complements roadmap **LLM-assisted trace analysis** (post-mortem) with cheap always-on counters.
- Pairs with **feature-service-catalog-context-hydration** and **Dynamic onboarding artifacts**.

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

## Acceptance criteria

- [ ] Terminal tasks include `context_quality` in API responses when execution reached RUNNING.
- [ ] Metrics visible in CloudWatch with `repo` and `workflow_ref` dimensions.
- [ ] Documented interpretation guide in `EVALUATION.md`.
- [ ] Unit tests for aggregation logic in agent and handler layers.

Contributor guide

Open the contributing guide

Research direction

Start by locating the agent progress_writer or runner, the orchestrator task record and terminal TaskEvent handling, and the operator dashboard and CloudWatch metric entry points. Trace how completion data reaches API responses, then review EVALUATION.md and existing handler and agent tests. Done means the listed acceptance criteria are covered across aggregation, persistence, metrics, dashboard visibility, documentation, and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
backend-api-design, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.