redhat-et / redhat-et/ProtoBot
Add complexity/label-based model or effort tiering for the code agent in .fullsend/config.yaml
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5
- Forks
- 6
- Avg merge
- 23h 24m
- Merged PRs (30d)
- 66
Description
What happened
Issue #131 (https://github.com/redhat-et/ProtoBot/issues/131) requested a purely mechanical, 7-line documentation fix (adding a Contents/TOC block to one ADR file, mirroring an existing sibling format). Triage labeled it documentation and confirmed it was low-complexity. The code agent run (https://github.com/redhat-et/ProtoBot/actions/runs/35376105082) nonetheless ran at high effort on xai/grok-4.6 per .fullsend/config.yaml, costing $2.66 and taking ~12 minutes — about 18x the triage run's cost ($0.15) and roughly 2x the review run's cost ($1.26), for a change that ultimately touched a single file with no logic, interfaces, or cross-document consistency at stake. Inspection of .fullsend/config.yaml confirms the code agent's model is hardcoded (xai/grok-4.6) with no branch based on issue labels, estimated diff size, or triage-assigned complexity.
What could go better
This is one data point, but the root cause is structural and directly observable in the config, not incidental: there is currently no mechanism in .fullsend/config.yaml to scale the code agent's model or effort down for issues that triage has already identified as low-complexity (e.g., via the documentation label or an explicit complexity signal). This means every documentation-only or trivially-scoped issue pays the same premium-model cost as a substantive code change. I'm confident the cost skew is real (it's a direct dollar comparison from the run metadata); I'm less certain what the ideal tiering rule should be (label-based vs. triage-emitted complexity score vs. diff-size heuristic post-hoc) — that design choice is left to the implementer.
Proposed change
In redhat-et/ProtoBot's .fullsend/config.yaml, add a routing rule so the code stage selects a lower-cost model/effort tier (e.g., google-vertex/gemini-3.8-flash, already used elsewhere in this config for fix/prioritize) when the triggering issue carries the documentation label (or a future triage-emitted complexity signal), falling back to the current high-effort grok-4.6 tier for everything else. Keep the review stage's existing model mix unchanged since it already varies by sub-agent role.
Validation criteria
Over the next 10 code-agent runs on issues labeled documentation, average per-run cost should drop measurably (e.g., below ~$0.75, roughly in line with the review-agent cost for this trivial change) without any increase in review-agent-requested-changes or human-requested-changes rate on those same PRs compared to the current baseline.
Generated by retro agent from https://github.com/redhat-et/ProtoBot/pull/146
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading .fullsend/config.yaml, especially the code stage and the existing model choices used by fix/prioritize; review the referenced issue #131 and run metadata for the baseline. Done means documentation-labeled runs use a lower-cost code-agent tier, other runs retain the current tier, review settings remain unchanged, and the next 10 documentation runs show lower average cost without higher change-request rates.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai-infra-agents, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100