chaoss / chaoss/CollectOSS

collect_github_messages wildly inefficient with API calls for large repos

Open
#419 0 comments 0 reactions 0 assignees View on GitHub
tech debt
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

How did this take so long to identify...

collect_github_messages decides whether a repo is "small" (by checking if `issue_pr_sum` < 10) and then switches how it collects messages (comments i think)

https://github.com/chaoss/CollectOSS/blob/e2e751333eae020c541778b0c964c8b4fc9c7bf4/collectoss/tasks/github/messages.py#L42-L52

The "fast" path uses a call to `/repos/{owner}/{repo}/issues/comments` with the `since=` parameter

the slow path does an individual call per issue or comment:

https://github.com/chaoss/CollectOSS/blob/e2e751333eae020c541778b0c964c8b4fc9c7bf4/collectoss/tasks/github/messages.py#L123-L126

additionally, according to Sonnet 4.6 medium:

> The large repo path filters by parent issue/PR updated_at in the DB query
>
> The unified endpoint supports ?since= which filters by comment's own updated_at.
>
> These semantics differ: if someone changes a label on a 3-year-old issue (updating the issue's updated_at), the large repo path fetches ALL of that issue's old comments again. The unified endpoint with since would correctly fetch only comments that were actually new or edited since the cutoff.
>
> The large repo approach is actually less correct for incremental collection, not more correct — it re-fetches stale comment data that's already in the DB whenever a parent issue is touched for any reason.

This is very likely significantly contributing to #391

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.