collect_github_messages wildly inefficient with API calls for large repos
- Dominant language
- Python
- Stars
- 13
- Forks
- 17
- Avg merge
- 6h 59m
- Merged PRs (30d)
- 1
Description
How did this take so long to identify...
collect_github_messages decides whether a repo is "small" (by checking if `issue_pr_sum` < 10) and then switches how it collects messages (comments i think)
https://github.com/chaoss/CollectOSS/blob/e2e751333eae020c541778b0c964c8b4fc9c7bf4/collectoss/tasks/github/messages.py#L42-L52
The "fast" path uses a call to `/repos/{owner}/{repo}/issues/comments` with the `since=` parameter
the slow path does an individual call per issue or comment:
https://github.com/chaoss/CollectOSS/blob/e2e751333eae020c541778b0c964c8b4fc9c7bf4/collectoss/tasks/github/messages.py#L123-L126
additionally, according to Sonnet 4.6 medium:
> The large repo path filters by parent issue/PR updated_at in the DB query
>
> The unified endpoint supports ?since= which filters by comment's own updated_at.
>
> These semantics differ: if someone changes a label on a 3-year-old issue (updating the issue's updated_at), the large repo path fetches ALL of that issue's old comments again. The unified endpoint with since would correctly fetch only comments that were actually new or edited since the cutoff.
>
> The large repo approach is actually less correct for incremental collection, not more correct — it re-fetches stale comment data that's already in the DB whenever a parent issue is touched for any reason.
This is very likely significantly contributing to #391
Contributor guide
Assessment
This issue has not been assessed yet.