Plan: submit microdf to the Journal of Open Source Software

Plan for submitting microdf to the Journal of Open Source Software, following the pattern we used for policyengine.py (paper at recommend-accept in JOSS review, both reviewers signed off).

Read this first: the scope risk is the whole decision

JOSS's submission guidance explicitly excludes "minor utility packages" and "thin" wrappers, and the review criteria require "substantial scholarly effort". microdf is ~1,850 lines across two classes (MicroSeries, MicroDataFrame) that add weighting to pandas. On code volume alone an editor could reject it at pre-review as a utility library, and that outcome is more likely than not if the paper is written as "weighted pandas".

The submission is winnable, but only if the paper is framed around the methodological content rather than the wrapper: survey-weighted quantiles by inverse CDF (matching survey::svyquantile), frequency-weighted variance, Lorenz-curve Gini, top/bottom share estimation with proportional tie handling at the cutoff, and weighted poverty measures (rate, gap, deep gap, squared gap / FGT). That is a small library of survey-statistics implementations that PolicyEngine's published estimates depend on — not a convenience wrapper.

Recommendation: treat this as a go/no-go on a standalone submission, decided before anyone writes prose. If we go, the paper must lead with the survey-statistics methods above; a paper framed as "weighted pandas" is not worth drafting. microdf's impact evidence is unusually strong for a package this size (below), which is the strongest counterweight to the scope objection.

I'd put the go/no-go to Max before starting the paper.

Impact evidence we can point to (JOSS's strongest gate, and microdf passes it)

• Public repository since June 2018, 750 commits, 9 releases, 9 contributors — comfortably clears "6+ months of public history with sustained development", and no repo-dump problem.
• Depended on by policyengine-us and policyengine-uk (both in pyproject.toml), plus policyengine-data and a long tail of study repos (free-school-meals, uk-extended-childcare, electricity-vat-cut, young-worker-nics, child-poverty-impact-dashboard, …).
• Therefore in the dependency path of every published PolicyEngine distributional estimate — poverty rates, deciles, Gini — which is exactly the "integration into research workflows" JOSS asks for.
• Add PyPI download statistics and any external (non-PolicyEngine) dependents at submission time.

What we reuse from policyengine.py

That repo is the template — copy the structure, not the prose:

• paper.md — YAML header (title, tags, authors with ORCIDs, affiliations, bibliography: paper.bib), then # Summary, # Statement of Need, # State of the Field, # Software Design, # Research Impact Statement, # Acknowledgements, # AI Usage Disclosure, # References. ~1,700 words with a comparison table. The Research Impact Statement and AI Usage Disclosure sections are both required by current JOSS guidance — easy to miss.
• paper.bib — 36 entries; the survey-statistics and PolicyEngine ones carry over directly.
• .github/workflows/draft-pdf.yml — openjournals/openjournals-draft-action@master with journal: joss, triggered on changes to paper.md/paper.bib/figures, uploading paper.pdf as an artifact. Copy verbatim.
• CITATION.cff, CODE_OF_CONDUCT.md — both present there, both missing here.

Checklist

1. Repository must-haves (JOSS reviewer checklist)

• CITATION.cff — absent
• CODE_OF_CONDUCT.md — absent
• MIT LICENSE — present ✅
• CONTRIBUTING.md — present ✅ (.github/CONTRIBUTING.md)
• Automated tests + CI — present ✅ (62 tests, Python 3.9–3.13 matrix)
• Fix CI on the default branch. .github/workflows/master.yml triggers on push: branches: [master], but the default branch is main — so the CI and docs-deploy jobs have not run on the default branch since the rename. A reviewer checking "tests run on the main branch" will see nothing.
• Fix the docs build. master.yml runs jb build docs/. and installs a [dev,docs] extra that doesn't exist in pyproject.toml, while pr.yaml uses myst build. docs/_toc.yml (Jupyter Book 1) and docs/myst.yml (MyST/JB2) are both present and disagree. Pick MyST, delete _toc.yml, and confirm a published docs site — JOSS wants installation instructions, example usage, and API documentation reachable by a reviewer.
• API documentation — currently there is no rendered API reference, only docs/home.md, docs/examples.md, and one Gini notebook. This is the largest documentation gap for review.
• Statement of need in the README (JOSS checks the repo, not just the paper)

2. Correctness before review, not during

A reviewer will run the statistics. The open correctness issues should be closed first — issue 300 (weights silently dropped by non-overridden pandas methods), issue 301, issue 302 (NaN handling in quantile/median), issue 303, issue 304, plus issue 179 (gini with duplicated index) and issue 291 (unweighted cov/corr). PRs 307-313 cover most of these. Finding a wrong weighted median mid-review is the worst possible time to find it.

3. Paper

• paper.md + paper.bib at the repo root, drafted around the survey-statistics framing above
• Author list and ORCIDs agreed (Max as the principal author of the statistical implementations; whoever else has contributed substantively)
• # State of the Field: compare against samplics, statsmodels' weighted statistics, R's survey package, and plain pandas + manual weighting — a table, as in the policyengine.py paper
• # Research Impact Statement: the dependency evidence above, with download numbers
• # AI Usage Disclosure
• Copy draft-pdf.yml and check the rendered PDF before submitting

4. Submit

• Tagged release + Zenodo deposit → DOI (JOSS requires the archive at acceptance; doing it early is harmless)
• Submit at joss.theoj.org, respond in the pre-review issue
• Reviewer response window is 2 weeks; substantive changes 4–6 weeks

Sequencing

Merge the open correctness PRs → fix CI-on-main and the docs build → add CITATION.cff/CODE_OF_CONDUCT.md → add an API reference → then draft the paper. The paper is last: it should describe a repository a reviewer can actually check out and run.

Sources: JOSS submission guide, JOSS review criteria, and PolicyEngine/policyengine.py (paper.md, paper.bib, .github/workflows/draft-pdf.yml).

Start by reading the linked JOSS submission and review guidance, then compare PolicyEngine/policyengine.py files such as paper.md, paper.bib, and .github/workflows/draft-pdf.yml with this repository. Inspect .github/workflows/master.yml, the docs configuration, and the listed correctness issues before proposing a sequence. Done means the maintainer has made the go/no-go decision and the repository, documentation, and paper are ready for submission.