yakew7 / yakew7/Fair-Code

All 7 audit READMEs' reproducibility checklists claim a stratified train/test split that no unfair.py/fair.py script performs

Open
#678 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug documentation
Dominant language
HTML
Stars
47
Forks
45
Avg merge
1h 57m
Merged PRs (30d)
95

Description

Where: COMPAS/README.md:9, German Credit Lending/README.md:9, AI Fair Recruitment/README.md:9, Insurance Denial/README.md:9, Benefits Denial/README.md:9, Healthcare Readmission/README.md:9, Tenant Screening/README.md:9 - each reads:

- [ ] Split: 80/20 train/test, stratified (`test_size: 0.2`)

The gap: contradicted by all 14 train_test_split(...) calls across the 7 audit folders' own unfair.py/fair.py scripts - none of them pass stratify=. Only faircode/benchmark.py's unified harness actually stratifies (train_test_split(..., stratify=y if y.nunique() > 1 else None)); none of the 7 standalone audit scripts that these checklists document do.

Repro:

$ grep -n "stratified" */README.md
COMPAS/README.md:9:- [ ] Split: 80/20 train/test, stratified (`test_size: 0.2`)
... (7 hits total, one per audit README)

$ grep -n "stratify" */unfair.py */fair.py
(0 hits)

Fix direction: either add stratify=y to all 14 train_test_split calls (making the scripts match the documented checklist), or remove "stratified" from all 7 checklists (making the docs match the actual scripts). Since this is the same wording copy-pasted across all 7 audit READMEs, fix all 7 (or all 14 scripts) in the same pass.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the seven README.md files at the listed checklist lines, then inspect the 14 train_test_split calls in each audit folder’s unfair.py and fair.py. Use the grep commands in the issue to verify the mismatch and confirm with maintainers whether the scripts or checklists should be authoritative. Done means all seven documented checklists accurately describe the corresponding standalone scripts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
documentation, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.