redhat-et / redhat-et/ProtoBot

Add complexity/label-based model or effort tiering for the code agent in .fullsend/config.yaml

Open
#149 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

blocked infra ready-for-triage
Dominant language
Go
Stars
5
Forks
6
Avg merge
23h 24m
Merged PRs (30d)
66

Description

What happened

Issue #131 (https://github.com/redhat-et/ProtoBot/issues/131) requested a purely mechanical, 7-line documentation fix (adding a Contents/TOC block to one ADR file, mirroring an existing sibling format). Triage labeled it documentation and confirmed it was low-complexity. The code agent run (https://github.com/redhat-et/ProtoBot/actions/runs/35376105082) nonetheless ran at high effort on xai/grok-4.6 per .fullsend/config.yaml, costing $2.66 and taking ~12 minutes — about 18x the triage run's cost ($0.15) and roughly 2x the review run's cost ($1.26), for a change that ultimately touched a single file with no logic, interfaces, or cross-document consistency at stake. Inspection of .fullsend/config.yaml confirms the code agent's model is hardcoded (xai/grok-4.6) with no branch based on issue labels, estimated diff size, or triage-assigned complexity.

What could go better

This is one data point, but the root cause is structural and directly observable in the config, not incidental: there is currently no mechanism in .fullsend/config.yaml to scale the code agent's model or effort down for issues that triage has already identified as low-complexity (e.g., via the documentation label or an explicit complexity signal). This means every documentation-only or trivially-scoped issue pays the same premium-model cost as a substantive code change. I'm confident the cost skew is real (it's a direct dollar comparison from the run metadata); I'm less certain what the ideal tiering rule should be (label-based vs. triage-emitted complexity score vs. diff-size heuristic post-hoc) — that design choice is left to the implementer.

Proposed change

In redhat-et/ProtoBot's .fullsend/config.yaml, add a routing rule so the code stage selects a lower-cost model/effort tier (e.g., google-vertex/gemini-3.8-flash, already used elsewhere in this config for fix/prioritize) when the triggering issue carries the documentation label (or a future triage-emitted complexity signal), falling back to the current high-effort grok-4.6 tier for everything else. Keep the review stage's existing model mix unchanged since it already varies by sub-agent role.

Validation criteria

Over the next 10 code-agent runs on issues labeled documentation, average per-run cost should drop measurably (e.g., below ~$0.75, roughly in line with the review-agent cost for this trivial change) without any increase in review-agent-requested-changes or human-requested-changes rate on those same PRs compared to the current baseline.


Generated by retro agent from https://github.com/redhat-et/ProtoBot/pull/146

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading .fullsend/config.yaml, especially the code stage and the existing model choices used by fix/prioritize; review the referenced issue #131 and run metadata for the baseline. Done means documentation-labeled runs use a lower-cost code-agent tier, other runs retain the current tier, review settings remain unchanged, and the next 10 documentation runs show lower average cost without higher change-request rates.

Written by the indexing model from the issue text.

Assessment

Domain
ai-infra-agents, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.