libgit2 / libgit2/libgit2sharp

RewriteHistory extremely slow

Open
#1,684 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
3.5k
Forks
925
PR merge metrics
No merged PRs in 30d

Description

Reproduction steps

Find a repo with 3000+ commits
Run the below code against that repo to rewrite the repo history:

private static void RewriteHistory(string repoPath, ICollection<string> subFolders)
{
    using (var repo = new Repository(repoPath))
    {
        var refs = repo.Refs.Where(r => r.CanonicalName.EndsWith("develop")).OrderBy(r => r.CanonicalName).ToList();
        var commits = repo.Commits.QueryBy(new CommitFilter { IncludeReachableFrom = refs }).ToArray();
        var count = 0;
        var stopwatch = Stopwatch.StartNew();

        Logger.InfoFormat("Found {0} commits", commits.Length);
        Logger.InfoFormat("Rewrite history started");

        repo.Refs.RewriteHistory(new RewriteHistoryOptions
        {
            OnError = OnError,
            OnSucceeding = OnSucceeding,
            PruneEmptyCommits = true,
            CommitTreeRewriter = commit =>
            {
                Logger.InfoFormat("Commit {0}/{1}", ++count, commits.Length);

                var tree = TreeDefinition.From(commit);
                foreach (var treeEntry in commit.Tree)
                {
                    var match = subFolders.FirstOrDefault(s => treeEntry.Path.StartsWithIgnoreCase(s));

                    if (match == null)
                        tree.Remove(treeEntry.Path);
                }

                return tree;
            }
        }, commits);
        Logger.InfoFormat("Rewrite history ended. Took {0}", stopwatch.Elapsed);

        repo.Reset(ResetMode.Hard, repo.Head.Commits.First());
    }
}
Expected behavior

This should take a reasonable time. I expect it to take a few minutes for sure but not hours. For example when running something similar in git from the command like it takes minutes. I'm sorry I don't have specific times but I can try to provide more data to compare performance if necessary.

Actual behavior

This takes 1h 43 minutes of which 1h 40 to just start the rewrite and about a couple of minutes to go through all the commits.
INFO : Found 3486 commits
INFO : Rewrite history started
[between these two log messages there is a 1h 40m gap]
INFO : Commit 1/3486
INFO : Commit 2/3486
...
INFO : Rewrite history ended. Took 01:43:51.5536779

Version of LibGit2Sharp (release number or SHA1)

0.26.0 + 2.0.267 (native)

Operating system(s) tested; .NET runtime tested

.net 4.6.1 - Win 2012R2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the report with a repository containing 3,000+ commits and measure the delay between “Rewrite history started” and the first CommitTreeRewriter callback. Inspect the RewriteHistory call and its CommitTreeRewriter setup shown in the issue. Done means the same history rewrite completes in minutes rather than hours while retaining the reported pruning behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, git
Domain
devtools, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.