github-community-projects / github-community-projects/pr-conflict-detector

Feature: Reduce API usage by cloning repos and analyzing diffs locally

Open
#5 1 comment 0 reactions 0 assignees View on GitHub
enhancement keep
Dominant language
Python
Stars
4
Forks
1
Avg merge
1h 13m
Merged PRs (30d)
12

Description

## Problem

The current approach makes one API call per open PR to fetch its changed files and patch data (`GET /repos/{owner}/{repo}/pulls/{number}/files`). For a repo with 500 open PRs, that's 500 API calls just for file data — before any verification calls. Across an organization with dozens of active repos, this adds up quickly and can hit rate limits (5,000 requests/hour for PATs, 15,000 for GitHub App installations).

## Proposed Approach

Instead of fetching file diffs via the API, clone the repository locally and use `git diff` to compute changed files and line ranges between each PR's head branch and the base branch.

### How It Would Work

1. **Shallow clone the repo** with all open PR head refs:
```bash
git clone --bare --filter=blob:none https://github.com/{owner}/{repo}.git
git fetch origin +refs/pull/*/head:refs/pull/*/head
```
Using a blobless clone (`--filter=blob:none`) keeps the initial clone fast — only tree objects are fetched eagerly, and blobs are fetched on demand when `git diff` needs them.

2. **Compute diffs locally** for each open PR:
```bash
git diff --name-only origin/main...refs/pull/{number}/head
git diff -U0 origin/main...refs/pull/{number}/head -- {file}
```
Parse the unified diff output to extract the same line range data we currently get from the API's patch field.

3. **Run conflict detection** using the same `find_file_overlaps()` algorithm — only the data source changes.

### API Savings

| Step | Current (API) | Proposed (Local) |
|------|--------------|-----------------|
| List open PRs | 1 call per 100 PRs | Same (still needed) |
| Fetch PR files/patches | **1 call per PR** | **0 calls** |
| Verify conflicts (optional) | 1 call per conflict pair | Same or could use local merge |
| **Total for 500 PRs** | **~506 calls** | **~6 calls** |

### Additional Benefits

- **True merge conflict detection**: With a local clone, we could run `git merge-tree` (Git 2.38+) to simulate merges without any API calls, replacing the current Phase 2 `VERIFY_CONFLICTS` API check. This would give us actual merge conflict detection rather than the heuristic-based overlap analysis.
```bash
git merge-tree --write-tree refs/pull/1/head refs/pull/2/head
# Exit code 1 = conflict, exit code 0 = clean merge
```
- **No rate limit concerns**: Once cloned, all analysis is CPU-bound, not API-bound.
- **Richer diff data**: Local diffs give full context, not just the first 300 files (API pagination limit per PR).

### Trade-offs to Consider

- **Disk space**: A blobless clone of a large monorepo could still be significant. Should be cleaned up after the run.
- **Clone time**: Initial clone + ref fetch takes time, especially for large repos. For small repos with few PRs, the API approach may actually be faster. This argues for making it configurable or auto-selecting based on PR count.
- **Git dependency**: Requires `git` to be available in the container (already present in the Docker image's base `python:3.14-slim`, but would need `git` installed).
- **GitHub Actions disk limits**: Runners have ~14GB of free space. Most repos will be fine, but extremely large monorepos could be tight with blobless clones.

## Configuration

```yaml
ANALYSIS_MODE: "local" # "api" (default, current behavior) or "local" (clone-based)
```

Could also support `"auto"` which uses the API for repos with <50 open PRs and local clone for larger ones.

## Acceptance Criteria

- [ ] Local clone mode fetches all open PR head refs efficiently
- [ ] Diff parsing produces identical results to the current API-based approach
- [ ] API call count is reduced to only what's needed (list PRs, create issues)
- [ ] Disk is cleaned up after each repo is processed
- [ ] Falls back to API mode gracefully if clone fails
- [ ] Performance benchmarks comparing API vs local for small/medium/large repos
- [ ] Documentation updated with new config option

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing find_file_overlaps algorithm and the Phase 2 VERIFY_CONFLICTS flow, then trace how API file and patch data enters conflict analysis. Compare that path with the proposed ANALYSIS_MODE configuration and define completion against the clone, diff parsing, cleanup, fallback, benchmarking, and documentation criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github-actions, python
Domain
ci-cd, devops, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.