neulab / neulab/agent-data-protocol
Add a check for stale generated sample fixtures
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 193
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Several checked-in sample_std.json fixtures drifted from the current converter output even when the semantic conversion was sound. A repository check should catch this earlier.
Acceptance criteria:
- Add a script or test that regenerates committed samples from
sample_raw.json/sample_atif.jsonand reports drift. - Make the check usable locally and in CI.
- Keep the check deterministic and skip datasets with unavailable optional dependencies in a clearly reported way.
- Document how to refresh fixtures when intentional converter changes occur.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the converter and the committed sample_raw.json, sample_atif.json, and sample_std.json fixtures, then run the existing sample-generation or test workflow. Add a deterministic local and CI check that reports drift, clearly skips unavailable optional dependencies, and document the intentional fixture-refresh process. Done means stale fixtures fail the check while valid optional-dependency skips are visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100