HexHive / HexHive/pubstats

Issues found while automatically reviewing the repo

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
35
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I asked GPT 5.6 Sol (High) to review the codebase for PL:

> Please make a critical review of this repo's logic for aggregating "top author" information. There were many inaccuracies in the past, as you can see in the commit history, so there are likely more. For now, just focus on the PL field

It found one critical problem and some smaller areas for improvement (see below). I will open a PR for the critical problem first. If such AI-enabled contributions are not welcome, let me know and I'll drop the PR. Otherwise, I can also open PRs for the three other problems, which I agree it would be good to address.

---

I found four concrete PL accuracy problems. The first is severe and currently removes an entire venue’s recent output.

1. **[P1] PLDI has been almost entirely omitted since 2023.**
PACMPL now encodes PLDI in its `number` field, but the allowlist lacks `"PLDI"` at [pubs.py:50](pubstats/pubs.py:50). The current DBLP snapshot contains:

- 83 PLDI records for 2023
- 89 for 2024
- 88 for 2025

All 260 are rejected. The pickle consequently reports 0 PLDI papers in 2023–24 and only one unusually encoded record in 2025. This is the same class of problem fixed for `OOPSLA1`/`OOPSLA2` in commit `37b55b3`.

::code-comment{title="[P1] PACMPL allowlist omits PLDI" body="Since 2023, DBLP represents PLDI papers as journal='Proc. ACM Program. Lang.' with number='PLDI'. Because PLDI is absent here, the current snapshot silently drops 83 papers from 2023, 89 from 2024, and 88 from 2025. Add PLDI and preferably validate or warn on previously unseen PACMPL issue labels." file="pubstats/pubs.py" start=50 priority=1}

2. **[P2] Rankings mix incomplete years without indicating that they are provisional.**
`current_year` is simply the largest observed year at [top_authors.py:90](pubstats/top_authors.py:90). The present dataset therefore ranks authors using 91 POPL 2026 papers while containing no 2026 ICFP, OOPSLA, or PLDI papers. It also defines “last five years” as 2022–26, dropping the complete 2021 year in favor of an incomplete 2026. This systematically favors authors publishing in venues already ingested.

::code-comment{title="[P2] Partial years are treated as complete" body="The maximum observed publication year becomes the ranking year and recent-window endpoint, even when only one PL venue has been ingested for that year. The current snapshot includes POPL 2026 but no 2026 ICFP/OOPSLA/PLDI, so totals and Rel5 are venue-biased and 2021 is prematurely excluded. Use an explicit last-complete year or clearly separate provisional results." file="pubstats/top_authors.py" start=90 priority=2}

3. **[P2] The six-page heuristic still counts non-paper proceedings material.**
For records with pages, length is effectively the only content filter at [parse_dblp.py:181](pubstats/parse_dblp.py:181). The PL pickle includes numerous panels and tutorials, despite the README claiming these are removed. Examples include:

- “Tutorial Notes on Partial Evaluation,” POPL 1993
- “How to Get a Paper Accepted at OOPSLA (Panel),” OOPSLA 1993
- at least 13 explicitly titled OOPSLA panels

This changes eligibility and totals: Grady Booch has four credited publications, two of which are panels; Randall B. Smith reaches the three-publication cutoff only because a panel is counted.

::code-comment{title="[P2] Long panels and tutorials count as papers" body="Any matching venue record of at least six pages is admitted, so long panels, tutorials, debates, and similar proceedings material receive full publication credit. The existing title exclusions only run when pages are missing. PL contains concrete ranking-changing examples; selection needs DBLP record-type/key rules or a maintained exclusion policy rather than page length alone." file="pubstats/parse_dblp.py" start=181 priority=2}

4. **[P2] The displayed median author counts are not actually medians.**
Both `(A)` and `(A5)` round the median to an integer at [top_authors.py:152](pubstats/top_authors.py:152) and [top_authors.py:166](pubstats/top_authors.py:166). Python’s banker’s rounding makes this particularly misleading: 2.5 displays as 2 and 4.5 as 4. In the current PL results, 269 ranked authors have half-integer all-time medians; Philip Wadler’s 2.5 is shown as 2, for example.

::code-comment{title="[P2] Median values are rounded incorrectly" body="The template says this column is the median, but round() discards valid half values and uses banker's rounding: 2.5 becomes 2 and 4.5 becomes 4. Preserve the median, formatting integers without a decimal and half-integers with .5; apply the same correction to med5." file="pubstats/top_authors.py" start=152 priority=2}

There are no automated tests or venue/year sanity checks. In particular, a simple assertion or warning on abrupt per-venue disappearance would likely have caught the PLDI regression immediately. I made no repository changes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the cited logic in pubstats/pubs.py, pubstats/top_authors.py, and pubstats/parse_dblp.py, then inspect the current PL data and commit 37b55b3. Done means PLDI records are retained, incomplete years and non-paper material are handled as specified, median values remain accurate, and venue/year sanity checks cover regressions; no automated tests currently exist.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.