scripts/freeze_paper_results.py's generated MANIFEST.md and docstring still say "what a paper should cite," contradicting the lifted freeze
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 47
- Forks
- 45
- Avg merge
- 1h 57m
- Merged PRs (30d)
- 95
Description
Where: scripts/freeze_paper_results.py:1 (module docstring: "Freeze the current results/ into paper/results-frozen/ for a paper citation") and line 110 (hardcoded into every generated MANIFEST.md: "This snapshot is what a paper should cite. \results/` at the repo root keeps changing as"`).
The gap: paper/results-frozen/MANIFEST.md (real, checked-in file) opens with exactly that sentence and has no disclaimer anywhere. This contradicts CLAUDE.md (repo policy, checked in): "paper/results-frozen/ ... is no longer under any special protection ... not the evidence backing a live citation." README.md's own "Reproducibility & Results History" section already correctly says "It's kept as historical reference, not as evidence for a live publication." A whole sweep of commits fixed this exact stale framing in README/CONTRIBUTING/CONTRIBUTORS/ROADMAP/CHANGELOG/website/llms.txt/faircode/benchmark.py when the freeze was lifted - scripts/freeze_paper_results.py was never touched by any of them, so it still generates a MANIFEST.md that reads like the freeze is still in force.
Repro:
$ head -5 scripts/freeze_paper_results.py
"""Freeze the current results/ into paper/results-frozen/ for a paper citation.
...
$ head -5 paper/results-frozen/MANIFEST.md
# Frozen results provenance
This snapshot is what a paper should cite. `results/` at the repo root keeps changing as
...
$ grep -n "no longer under any special protection\|not the evidence" CLAUDE.md
19:`paper/results-frozen/` is kept in the repo, but it is no longer under any special protection - it's
21:year's real paper run, not the evidence backing a live citation.
tests/test_freeze_paper_results.py's own docstring confirms this exact text is untested (only mirror_for_mcp is in scope there).
Fix direction: reword the module docstring and the generated MANIFEST.md text to match CLAUDE.md/README's current framing - a comparison snapshot for next year's paper run, not evidence for a live citation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the module docstring and MANIFEST.md generation around line 110 in scripts/freeze_paper_results.py, then compare the wording with CLAUDE.md and README.md. Update the stale framing in the generator and checked-in paper/results-frozen/MANIFEST.md, and inspect tests/test_freeze_paper_results.py to confirm the affected text is not currently covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100