yakew7 / yakew7/Fair-Code

scripts/freeze_paper_results.py's generated MANIFEST.md and docstring still say "what a paper should cite," contradicting the lifted freeze

Open Beginner friendly
#680 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug documentation
Dominant language
HTML
Stars
47
Forks
45
Avg merge
1h 57m
Merged PRs (30d)
95

Description

Where: scripts/freeze_paper_results.py:1 (module docstring: "Freeze the current results/ into paper/results-frozen/ for a paper citation") and line 110 (hardcoded into every generated MANIFEST.md: "This snapshot is what a paper should cite. \results/` at the repo root keeps changing as"`).

The gap: paper/results-frozen/MANIFEST.md (real, checked-in file) opens with exactly that sentence and has no disclaimer anywhere. This contradicts CLAUDE.md (repo policy, checked in): "paper/results-frozen/ ... is no longer under any special protection ... not the evidence backing a live citation." README.md's own "Reproducibility & Results History" section already correctly says "It's kept as historical reference, not as evidence for a live publication." A whole sweep of commits fixed this exact stale framing in README/CONTRIBUTING/CONTRIBUTORS/ROADMAP/CHANGELOG/website/llms.txt/faircode/benchmark.py when the freeze was lifted - scripts/freeze_paper_results.py was never touched by any of them, so it still generates a MANIFEST.md that reads like the freeze is still in force.

Repro:

$ head -5 scripts/freeze_paper_results.py
"""Freeze the current results/ into paper/results-frozen/ for a paper citation.
...
$ head -5 paper/results-frozen/MANIFEST.md
# Frozen results provenance

This snapshot is what a paper should cite. `results/` at the repo root keeps changing as
...
$ grep -n "no longer under any special protection\|not the evidence" CLAUDE.md
19:`paper/results-frozen/` is kept in the repo, but it is no longer under any special protection - it's
21:year's real paper run, not the evidence backing a live citation.

tests/test_freeze_paper_results.py's own docstring confirms this exact text is untested (only mirror_for_mcp is in scope there).

Fix direction: reword the module docstring and the generated MANIFEST.md text to match CLAUDE.md/README's current framing - a comparison snapshot for next year's paper run, not evidence for a live citation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the module docstring and MANIFEST.md generation around line 110 in scripts/freeze_paper_results.py, then compare the wording with CLAUDE.md and README.md. Update the stale framing in the generator and checked-in paper/results-frozen/MANIFEST.md, and inspect tests/test_freeze_paper_results.py to confirm the affected text is not currently covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.