neulab / neulab/agent-data-protocol

Add a check for stale generated sample fixtures

Open
#300 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
193
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Several checked-in sample_std.json fixtures drifted from the current converter output even when the semantic conversion was sound. A repository check should catch this earlier.

Acceptance criteria:

  • Add a script or test that regenerates committed samples from sample_raw.json/sample_atif.json and reports drift.
  • Make the check usable locally and in CI.
  • Keep the check deterministic and skip datasets with unavailable optional dependencies in a clearly reported way.
  • Document how to refresh fixtures when intentional converter changes occur.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the converter and the committed sample_raw.json, sample_atif.json, and sample_std.json fixtures, then run the existing sample-generation or test workflow. Add a deterministic local and CI check that reports drift, clearly skips unavailable optional dependencies, and document the intentional fixture-refresh process. Done means stale fixtures fail the check while valid optional-dependency skips are visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, documentation, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.