github-community-projects / github-community-projects/pr-conflict-detector
Feature: Reduce API usage by cloning repos and analyzing diffs locally
- Dominant language
- Python
- Stars
- 4
- Forks
- 1
- Avg merge
- 1h 13m
- Merged PRs (30d)
- 12
Description
## Problem
The current approach makes one API call per open PR to fetch its changed files and patch data (`GET /repos/{owner}/{repo}/pulls/{number}/files`). For a repo with 500 open PRs, that's 500 API calls just for file data — before any verification calls. Across an organization with dozens of active repos, this adds up quickly and can hit rate limits (5,000 requests/hour for PATs, 15,000 for GitHub App installations).
## Proposed Approach
Instead of fetching file diffs via the API, clone the repository locally and use `git diff` to compute changed files and line ranges between each PR's head branch and the base branch.
### How It Would Work
1. **Shallow clone the repo** with all open PR head refs:
```bash
git clone --bare --filter=blob:none https://github.com/{owner}/{repo}.git
git fetch origin +refs/pull/*/head:refs/pull/*/head
```
Using a blobless clone (`--filter=blob:none`) keeps the initial clone fast — only tree objects are fetched eagerly, and blobs are fetched on demand when `git diff` needs them.
2. **Compute diffs locally** for each open PR:
```bash
git diff --name-only origin/main...refs/pull/{number}/head
git diff -U0 origin/main...refs/pull/{number}/head -- {file}
```
Parse the unified diff output to extract the same line range data we currently get from the API's patch field.
3. **Run conflict detection** using the same `find_file_overlaps()` algorithm — only the data source changes.
### API Savings
| Step | Current (API) | Proposed (Local) |
|------|--------------|-----------------|
| List open PRs | 1 call per 100 PRs | Same (still needed) |
| Fetch PR files/patches | **1 call per PR** | **0 calls** |
| Verify conflicts (optional) | 1 call per conflict pair | Same or could use local merge |
| **Total for 500 PRs** | **~506 calls** | **~6 calls** |
### Additional Benefits
- **True merge conflict detection**: With a local clone, we could run `git merge-tree` (Git 2.38+) to simulate merges without any API calls, replacing the current Phase 2 `VERIFY_CONFLICTS` API check. This would give us actual merge conflict detection rather than the heuristic-based overlap analysis.
```bash
git merge-tree --write-tree refs/pull/1/head refs/pull/2/head
# Exit code 1 = conflict, exit code 0 = clean merge
```
- **No rate limit concerns**: Once cloned, all analysis is CPU-bound, not API-bound.
- **Richer diff data**: Local diffs give full context, not just the first 300 files (API pagination limit per PR).
### Trade-offs to Consider
- **Disk space**: A blobless clone of a large monorepo could still be significant. Should be cleaned up after the run.
- **Clone time**: Initial clone + ref fetch takes time, especially for large repos. For small repos with few PRs, the API approach may actually be faster. This argues for making it configurable or auto-selecting based on PR count.
- **Git dependency**: Requires `git` to be available in the container (already present in the Docker image's base `python:3.14-slim`, but would need `git` installed).
- **GitHub Actions disk limits**: Runners have ~14GB of free space. Most repos will be fine, but extremely large monorepos could be tight with blobless clones.
## Configuration
```yaml
ANALYSIS_MODE: "local" # "api" (default, current behavior) or "local" (clone-based)
```
Could also support `"auto"` which uses the API for repos with <50 open PRs and local clone for larger ones.
## Acceptance Criteria
- [ ] Local clone mode fetches all open PR head refs efficiently
- [ ] Diff parsing produces identical results to the current API-based approach
- [ ] API call count is reduced to only what's needed (list PRs, create issues)
- [ ] Disk is cleaned up after each repo is processed
- [ ] Falls back to API mode gracefully if clone fails
- [ ] Performance benchmarks comparing API vs local for small/medium/large repos
- [ ] Documentation updated with new config option
Contributor guide
Research direction
Start by locating the existing find_file_overlaps algorithm and the Phase 2 VERIFY_CONFLICTS flow, then trace how API file and patch data enters conflict analysis. Compare that path with the proposed ANALYSIS_MODE configuration and define completion against the clone, diff parsing, cleanup, fallback, benchmarking, and documentation criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github-actions, python
- Domain
- ci-cd, devops, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100