BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(holdout-evaluator): holdout score goes stale silently as stories ship — no trigger to re-run after N deliveries

Open
#795 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

A holdout evaluation score is recorded against a specific commit and then stays in STATE.md frontmatter as though current, with no mechanism marking it stale as further stories merge past that commit.

## Observed

HS-006 scored 0.895 at `f73676d` on 2026-07-12. Three stories shipped to `develop` after that evaluation and the score was never re-measured. It still reads as the project's holdout standing.

This is the second instance of the same shape on this project — an earlier occurrence was fixed once by re-running, but nothing was added to prevent recurrence, so it recurred.

## Why it matters

The score's whole purpose is a current quality signal, and staleness is invisible: the number looks identical whether measured one commit ago or twenty. A gate reading it cannot tell the difference.

## Suggested fix

Either record the evaluated commit alongside the score and have any consumer compare it against `develop` HEAD, or trigger a re-run when N stories have merged since the evaluated commit. The commit is already recorded; what is missing is anything that compares it.

Proposal from a session review on switchboard-blue (plugin 1.0.0-rc.24), committed at `177d921a` in `.factory/session-reviews/improvement-proposals-2026-08-28.md`.

Contributor guide

Open the contributing guide

Research direction

The holdout evaluator logic is likely in a module like `holdout_evaluator.rs`. Examine the STATE.md frontmatter format and the commit recording mechanism. The fix involves comparing the recorded commit hash with the current HEAD of the `develop` branch and triggering a re-evaluation after N merges. Start by reading the session review proposal at `.factory/session-reviews/improvement-proposals-2026-08-28.md` to understand the suggested implementation. Verify the change by checking that the score updates correctly after new stories merge.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, observability-sre, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.