chaoss / chaoss/CollectOSS

pull request files collection could be more efficient

Open
#420 1 comment 0 reactions 0 assignees View on GitHub
disclosed-ai task:secondary tech debt
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

We might be able to squeeze a lot more performance from our api calls in PR files collection

Sonnet 4.6 has identified that we are constructing one graphql call per pull request:

https://github.com/chaoss/CollectOSS/blob/96adf3a4d68725db21622673ee6613693c0f5ace/collectoss/tasks/github/pull_requests/files_model/core.py#L52-L69

Some of these may return very few files.

Other places in the code already batch multiple queries into one graphQL call:

https://github.com/chaoss/CollectOSS/blob/96adf3a4d68725db21622673ee6613693c0f5ace/collectoss/tasks/frontend.py#L207-L212

We should do the same in pull request files to help make things faster (less waiting for network calls)

Contributor guide

Open the contributing guide

Research direction

Start with collectoss/tasks/github/pull_requests/files_model/core.py at lines 52-69, then compare the batching approach in collectoss/tasks/frontend.py at lines 207-212. Confirm how pull request files are collected and identify the current per-pull-request GraphQL calls. Done means equivalent requests are batched to reduce network waits while preserving the collected files.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, python
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.