All 7 audit READMEs' reproducibility checklists claim a stratified train/test split that no unfair.py/fair.py script performs
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 47
- Forks
- 45
- Avg merge
- 1h 57m
- Merged PRs (30d)
- 95
Description
Where: COMPAS/README.md:9, German Credit Lending/README.md:9, AI Fair Recruitment/README.md:9, Insurance Denial/README.md:9, Benefits Denial/README.md:9, Healthcare Readmission/README.md:9, Tenant Screening/README.md:9 - each reads:
- [ ] Split: 80/20 train/test, stratified (`test_size: 0.2`)
The gap: contradicted by all 14 train_test_split(...) calls across the 7 audit folders' own unfair.py/fair.py scripts - none of them pass stratify=. Only faircode/benchmark.py's unified harness actually stratifies (train_test_split(..., stratify=y if y.nunique() > 1 else None)); none of the 7 standalone audit scripts that these checklists document do.
Repro:
$ grep -n "stratified" */README.md
COMPAS/README.md:9:- [ ] Split: 80/20 train/test, stratified (`test_size: 0.2`)
... (7 hits total, one per audit README)
$ grep -n "stratify" */unfair.py */fair.py
(0 hits)
Fix direction: either add stratify=y to all 14 train_test_split calls (making the scripts match the documented checklist), or remove "stratified" from all 7 checklists (making the docs match the actual scripts). Since this is the same wording copy-pasted across all 7 audit READMEs, fix all 7 (or all 14 scripts) in the same pass.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the seven README.md files at the listed checklist lines, then inspect the 14 train_test_split calls in each audit folder’s unfair.py and fair.py. Use the grep commands in the issue to verify the mismatch and confirm with maintainers whether the scripts or checklists should be authoritative. Done means all seven documented checklists accurately describe the corresponding standalone scripts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- documentation, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100