neurostuff / neurostuff/autonima

No record of what changed between config versions: derive a semantic version diff in the UI

Open
#64 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4
Forks
1
Avg merge
12h 56m
Merged PRs (30d)
2

Description

Nothing in a run records what changed between one config version and the next. Reconstructing it means diffing YAML by hand, and a YAML diff is a poor proxy for a semantic one: cosmetic edits show up as changes, while a criterion being silently reworded shows up the same as a criterion being added.

This bit us while writing up the validation work. The scope table comparing dementia S1/S2/S3 had to be rebuilt by diffing configs after the fact, and separately it took a dedicated audit to establish that v3-all_pmids.yaml and v3-all_pmids-multi_analysis.yaml differ by exactly one line which states a default the first already inherits — the run names implied a manipulation that did not exist.

Why the UI is the right home for this

A changes: field in the config would work but will not be written reliably, because the person editing a config is optimising for the run, not for a future reader. The webui already knows a run's lineage and holds both config.executed.yaml files, so it can derive the diff rather than depend on anyone recording it.

Sketch:

  • a version-diff view for any two runs of a project, working from config.executed.yaml rather than the source YAML, so it reflects what actually executed including inherited defaults
  • grouped by stage (search / screening / retrieval / coordinates / annotation / output), since that is how the pipeline is reasoned about
  • semantic rather than textual where cheap: separate "criterion added / removed / reworded" from formatting, and mark edits that are cache-neutral. stage_hash and stage_signature_payloads already know which changes affect execution — surfacing that distinction directly answers "will this re-run anything?", which is the question people actually have
  • an optional free-text rationale captured at run time, stored alongside the run rather than in the config, so the why can be recorded without editing the artifact the diff is computed from
Related

Adjacent to #58 (unknown config keys silently dropped) — both are cases where the config is the method but the tooling does not help you see what the config actually says.

Filed out of the validation-paper work, where a "still missing" note about this was tracked in the outline; moving it here since the fix belongs in the tool rather than in the paper.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the webui code that loads a project's runs and their config.executed.yaml files, then read the existing stage_hash and stage_signature_payloads handling. Define the version-diff view around stage grouping, semantic change categories, cache-neutral status, and run-level rationale storage; done means users can compare any two runs and understand what changed and whether execution is affected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, frontend, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.