andreagrandi / andreagrandi/draftomen

Re-validate stored enrichment work when a validator changes

Open
#486 0 comments 0 reactions 0 assignees View on GitHub
draftomen enhancement size: M
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 23m
Merged PRs (30d)
181

Description

## Problem

The durable set-enrichment work store derives `WorkState.COMPLETED` from the presence of a result artifact, and `WorkIdentity` keys on the contract version, prompt, schema and model hashes only. A change to a *validator* therefore leaves previously stored results in place: a resumed run reuses the old parse and reports it as current, so a validator fix silently appears to have no effect on any work that was already recorded.

Measured during the #472 paid verification: after relaxing the guide-claim card-reference binding, the harness re-run reused all 613 stored records and still reported the old verdict (0 of 7 required mechanics). Only removing the stored guide *result* artifact — keeping the durable response — made the service take its documented `UNVALIDATED` path, re-parse the already-paid response locally and change the verdict to 4 of 7 at zero additional cost.

## Proposed change

Make validator identity part of durability: include the version of the validator that produced a result in the work identity (or record it in the result artifact) so that a validator change re-derives the affected results from their durable responses instead of reusing stale parses. Re-derivation must never issue a replacement paid request, and unchanged validators must keep reusing stored results without re-parsing.

## Acceptance Criteria

- [ ] AC1: `tests/test_set_enrichment_work.py` verifies that stored work recorded under one validator version is not reported as current after the validator version changes, and is re-derivable from its stored response without a new request.
- [ ] AC2: `tests/test_set_enrichment.py` verifies a resumed run under a changed validator re-parses the durable response, records the new result and issues no additional completion call, while an unchanged validator still reuses the stored result without re-parsing.
- [ ] AC3: `tests/test_set_enrichment.py` verifies a validator change that flips an acceptance-relevant outcome (for example a formerly rejected claim becoming a retained uncertain claim) is reflected in the resumed run's result.

## Dependencies

None. Related: #472 (the verification that exposed this).

## Implementation classification

- **Estimated size:** M
- **Orchestration risk:** High
- **Reason:** Changes the durability contract and resumption semantics that every paid enrichment run depends on, so it needs explicit reuse, re-derivation and no-replacement-request coverage.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.