github / github/gh-aw

[copilot-opt] Reduce wasted PRs from operational-value-designer blocked-outcome recording

Open
#60,897 0 comments 0 reactions 0 assignees View on GitHub
cookie copilot-opt operational-value-designer optimization
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

### Problem
The `operational-value-designer` workflow is opening one standalone pull request per target workflow to record "blocked" grader-design outcomes, even when no grader can be implemented. This burns a full Copilot agent session per PR for a result that carries no merge value.

### Evidence
- Analysis window: 2026-08-31 to 2026-09-14 (14-day scope); PR dataset spans 2026-08-18 to 2026-09-14 (1000 most recent Copilot PRs)
- Sessions analyzed: 50 GitHub Actions runs in `sessions-list.json` (no conversation/events telemetry was available — see the companion telemetry-gap issue)
- Key metrics:
- 62 PRs in the dataset have titles matching `operational-value` / `operational value`; **60 of 62 (97%) are CLOSED without merging**, only 2 merged
- Representative titles: "Record blocked operational-value study for X", "[WIP] Design deterministic operational-value grader for Y", "operational-value-study: block copilot-opt (no defensible direct metric)"
- On 2026-09-04 alone, dozens of these PRs were opened and closed within minutes of each other (e.g. ``#58466``, ``#58468``, ``#58469``, ``#58470``, ``#58471`` all closed between 11:56 and 12:15 UTC), indicating a batch-dispatch pattern producing many near-simultaneous single-purpose PRs
- None of the sampled closed PRs (``#58541``, ``#58535``, ``#58466``, ``#58469``, ``#58468``) received any review comments before closing — they were self-closed by the automation with no human interaction

### Proposed Change
Change the `operational-value-designer` skill/workflow so that a "blocked" outcome is recorded directly (e.g. appended to a shared tracking file, issue, or dataset committed in one batched PR) instead of opening an individual pull request per workflow. Only open a PR when a grader is actually implementable and mergeable. If per-workflow PRs must continue, batch multiple blocked-outcome records into a single PR per run instead of one PR per workflow.

### Expected Impact
- Eliminates roughly 60 wasted full-agent-session PR cycles per ~3-week window
- Reduces reviewer/queue noise from self-closing, zero-comment PRs
- Frees agent session budget for higher-value grader implementations

### Notes
- Distinct root cause category: agent specialization/decomposition (task granularity mismatch — one session per near-trivial negative outcome)
- Data quality caveat: no `events.jsonl`/conversation logs were available for these sessions, so this finding is based on GitHub PR metadata correlation (Phase 3), not per-session tool traces (Phase 2)

> Generated by [⚡ Copilot Opt](https://github.com/github/gh-aw/actions/runs/34876283982) · copilot · auto · 85.8 AIC · ⌖ 10.5 AIC · ⊞ 8.5K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fcopilot-opt%22&type=issues)

Contributor guide

Open the contributing guide

Research direction

Start by locating the operational-value-designer skill/workflow and review the blocked-outcome behavior described in the issue, using sessions-list.json and the cited PR pattern as context. Define a batched recording path for blocked outcomes while preserving individual PRs for implementable graders. Done means blocked cases no longer create one standalone PR per workflow and implementable graders still produce mergeable PRs.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.