BOHICA-LABS / BOHICA-LABS/vsdd-factory
enhancement(holdout-evaluator): holdout score goes stale silently as stories ship — no trigger to re-run after N deliveries
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
A holdout evaluation score is recorded against a specific commit and then stays in STATE.md frontmatter as though current, with no mechanism marking it stale as further stories merge past that commit.
## Observed
HS-006 scored 0.895 at `f73676d` on 2026-07-12. Three stories shipped to `develop` after that evaluation and the score was never re-measured. It still reads as the project's holdout standing.
This is the second instance of the same shape on this project — an earlier occurrence was fixed once by re-running, but nothing was added to prevent recurrence, so it recurred.
## Why it matters
The score's whole purpose is a current quality signal, and staleness is invisible: the number looks identical whether measured one commit ago or twenty. A gate reading it cannot tell the difference.
## Suggested fix
Either record the evaluated commit alongside the score and have any consumer compare it against `develop` HEAD, or trigger a re-run when N stories have merged since the evaluated commit. The commit is already recorded; what is missing is anything that compares it.
Proposal from a session review on switchboard-blue (plugin 1.0.0-rc.24), committed at `177d921a` in `.factory/session-reviews/improvement-proposals-2026-08-28.md`.
Contributor guide
Research direction
The holdout evaluator logic is likely in a module like `holdout_evaluator.rs`. Examine the STATE.md frontmatter format and the commit recording mechanism. The fix involves comparing the recorded commit hash with the current HEAD of the `develop` branch and triggering a re-evaluation after N merges. Start by reading the session review proposal at `.factory/session-reviews/improvement-proposals-2026-08-28.md` to understand the suggested implementation. Verify the change by checking that the score updates correctly after new stories merge.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, observability-sre, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100