Comfy-Org / Comfy-Org/download-data

Bug: GitHub Releases API not paginated — older releases excluded from daily totals

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
4
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Summary
The fetcher calls the Releases API once without pagination, so only the first page (default 30 releases) is processed. Older releases beyond page 1 are ignored, which can undercount and make daily totals appear artificially stable.

Details
- Location: scripts/fetch-data.mjs:274–294 (fetchGitHubReleases)
- Current: GET https://api.github.com/repos/comfyanonymous/ComfyUI/releases (no per_page, no page loop)
- GitHub defaults per_page=30. Popular repos like ComfyUI have many releases; older ones still accrue downloads.
- Impact: Any downloads on older releases (outside first page) are omitted, skewing daily deltas. Over time, totals can plateau once interesting releases fall off page 1.

Proposed Fix
- Implement pagination:
- Request per_page=100
- Follow Link headers (or increment page starting from 1) until an empty page is returned
- Merge all releases before storing assets into asset_daily_stats
- Keep existing PAT/header logic. Consider short delays to respect rate limits.
- Log a concise summary: e.g., "Fetched releases across

pages".

Acceptance Criteria
- The fetcher retrieves >30 releases when the repo has them (verify by logs and snapshot counts)
- Daily deltas change to reflect contributions from older releases where appropriate
- No regression in DB schema or API endpoints

References
- scripts/fetch-data.mjs:274–294
- Delta computation: scripts/fetch-data.mjs:343–357

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.