libgit2 / libgit2/libgit2

In-memory diffs and merges slow on large repos

Open
#6,036 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
10.6k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

In-memory merges can be slow on a repository like https://github.com/mozilla/gecko-dev, e.g. several seconds to carry out a simple merge for a few changed files. This is around 500x slower than what's possible, comparing against a workaround (see benchmark at https://github.com/arxanas/git-branchless/commit/4c5740779a88059e2fa547bfac2b275cc886e869):

  • 1.9s for naive cherry-pick.
  • 2.8ms for workaround cherry-pick.

I believe this is because the in-memory Index structure always stores all files, even when the vast majority of them aren't changed. It would be best if the in-memory index could alternatively be backed by a tree + changed paths.

The workaround is as follows:

  • Find the commits to merge and calculate their merge-base, as appropriate.
  • Find all paths changed among each of those commits' trees compared to the merge-base.
  • Generate synthetic versions of each tree/commit only containing the changed paths.
  • Carry out the merge on the synthetic trees/commits.
  • Combine the result back with an original tree, overwriting any entries already in the tree. (It doesn't matter which tree, since the non-changed entries are all the same.)

Reference implementation for cherry-picking specifically: https://github.com/arxanas/git-branchless/blob/ec0d27427ab7a505d4109e4588e356d6a18da2fe/src/git/repo.rs#L726-L836

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked reference implementation in src/git/repo.rs, lines 726-836, and compare its synthetic-tree approach with libgit2's in-memory Index behavior. Benchmark large-repository merges such as the gecko-dev example; done means preserving merge results while substantially reducing the seconds-long runtime toward the referenced workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, git
Domain
devtools, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.