microsoft / microsoft/hve-core

fix(workflows): make backlog grooming resilient to malformed model output

Open
#2,827 0 comments 0 reactions 1 assignee Claimed by @rezatnoMsirhC View on GitHub
agents bug infrastructure instructions
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

Backlog Grooming Sweep validation exposed a series of related failures in which model-authored structural state, transport variance, or one candidate-local defect could terminate an otherwise valid shard and stop the full sweep. The first incidents appeared as incomplete `Superseded` lineage and summary counts that disagreed with final rows, but branch validation showed that these were symptoms of a broader contract-boundary problem.

The fix makes final candidate rows authoritative, derives structural state in trusted code, bounds narrowly recoverable representation variance, and carries safely attributable malformed rows as non-advisory `contract_error` evidence. Strict identity, provenance, partition, digest, and publication checks remain fail-closed.

## Observed Failure Modes

- `Superseded` selected without non-empty, distinct original-delivery and replacement-or-removal lineage.
- Model-authored assessed and deferred counts, stop reason, or cursor disagreeing with finalized rows.
- `Superseded` placed in `similarity_outcome` instead of the disposition field.
- The `deferred_reason` alias appearing instead of canonical `deferral_reason`.
- Safe-output transport escaping nested report JSON.
- Evidence strings and evidence arrays exceeding bounded output expectations.
- Public contributor issue data being rejected during candidate hydration.
- The reusable worker lacking pull-request read permission needed for lineage evidence.
- One safely attributable malformed row aborting its complete shard, checkpoint chain, reduction, and publication.

## Implemented Resolution

- The model publishes candidate rows only; trusted workflow code derives counts, cursor, bounded stop reason, timestamps, and inventory values.
- Candidate identity is bound to exactly one planned shard issue before advisory fields are validated.
- Complete, distinct supersession lineage remains mandatory; unsupported supersession claims are downgraded to `Uncertain` rather than accepted as invalid advice.
- Narrow, deterministic compatibility handling decodes transport escaping, canonicalizes the deferral-reason alias, bounds evidence, and records the exact issue #1946 field-placement normalization.
- Shard result, wave aggregate, checkpoint, and final aggregate contracts use v2 processing state with disjoint `Assessed`, `Deferred`, and `contract_error` issue sets.
- Contract-error diagnostics contain only fixed `{ issue, code }` records and cannot carry advisory model text.
- Recovery and final reduction preserve exact candidate coverage and provenance across all waves.
- Both tracker and history publishers independently require `contract_errors === 0` before mutation.
- Required contributor handling and pull-request read access are explicitly supported.

## Acceptance Criteria

- [x] `Superseded` is accepted only with non-empty, distinct original-delivery and replacement-or-removal evidence.
- [x] Structural run state is derived from validated final rows and trusted workflow inputs rather than duplicated model-authored summaries.
- [x] Narrow representation variance is normalized deterministically without introducing permissive JSON repair.
- [x] Safely attributable malformed rows are retained as fixed non-advisory contract errors while valid sibling rows continue.
- [x] Accepted rows, deferred rows, and contract-error issue IDs form an exact disjoint partition of planned candidates.
- [x] Checkpoint recovery and final reduction preserve contract errors, ordering, digests, and immutable provenance.
- [x] Tracker and history publication remain blocked whenever any contract error exists.
- [x] Source workflow, generated lock workflow, agent guidance, policy guidance, validator, and focused tests remain aligned.
- [x] A complete feature-branch sweep processed all 16 waves and produced a final aggregate with zero contract errors.
- [ ] Merge the implementation to `main` and complete a clean default-branch sweep and publication.

## Validation

- Worker and workflow contract suite: 86 passing tests.
- Wave validator suite: 49 passing tests.
- [Terminal feature-branch sweep run 34275535750](https://github.com/microsoft/hve-core/actions/runs/34275535750) completed final reduction and intentionally skipped successor dispatch.
- Final aggregate: 160 candidates accounted for, 154 assessed, 6 deferred, and 0 contract errors.

## Residual Risk

A syntax error in the single shard-wide nested `report-data` JSON string can still prevent parsing before candidate identity is bound. The validator intentionally does not guess how to repair malformed JSON. Per-candidate safe-output isolation is tracked separately in #2866 and does not block this implementation from merging.

## Related

- Original automated backlog grooming feature: #2558
- Consolidated summary-count incident: #2828
- Per-candidate safe-output hardening: #2866
- Initial supersession failure: https://github.com/microsoft/hve-core/actions/runs/33653607411
- Initial summary-count failure: https://github.com/microsoft/hve-core/actions/runs/33658405222
- Whole-envelope syntax failure: https://github.com/microsoft/hve-core/actions/runs/33838636334

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.