Comfy-Org / Comfy-Org/download-data
Bug: GitHub Releases API not paginated — older releases excluded from daily totals
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The fetcher calls the Releases API once without pagination, so only the first page (default 30 releases) is processed. Older releases beyond page 1 are ignored, which can undercount and make daily totals appear artificially stable.
Details
- Location: scripts/fetch-data.mjs:274–294 (fetchGitHubReleases)
- Current: GET https://api.github.com/repos/comfyanonymous/ComfyUI/releases (no per_page, no page loop)
- GitHub defaults per_page=30. Popular repos like ComfyUI have many releases; older ones still accrue downloads.
- Impact: Any downloads on older releases (outside first page) are omitted, skewing daily deltas. Over time, totals can plateau once interesting releases fall off page 1.
Proposed Fix
- Implement pagination:
- Request per_page=100
- Follow Link headers (or increment page starting from 1) until an empty page is returned
- Merge all releases before storing assets into asset_daily_stats
- Keep existing PAT/header logic. Consider short delays to respect rate limits.
- Log a concise summary: e.g., "Fetched releases across
pages".
Acceptance Criteria
- The fetcher retrieves >30 releases when the repo has them (verify by logs and snapshot counts)
- Daily deltas change to reflect contributions from older releases where appropriate
- No regression in DB schema or API endpoints
References
- scripts/fetch-data.mjs:274–294
- Delta computation: scripts/fetch-data.mjs:343–357
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.