libgit2 / libgit2/libgit2sharp

Performance regression when reading files from the tree in parallel

Open
#2,076 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
3.5k
Forks
925
PR merge metrics
No merged PRs in 30d

Description

Reproduction steps
  1. Add the following test cases in CommitFixture.cs:
[Fact]
public void CanReadCommit()
{
    var timer = new Stopwatch();
    var fileContents = new ConcurrentStack<string>();
    var path = SandboxStandardTestRepo();
    using (var repo = new Repository(path))
    {
        var latestCommit = repo.Head.Tip;
        var tree = latestCommit.Tree;
        timer.Start();
        for (var i = 0; i < 100000; i++)
        {
            fileContents.Push(ReadEntry("1.txt", tree));
            fileContents.Push(ReadEntry("README", tree));
            fileContents.Push(ReadEntry("new.txt", tree));
        }
        timer.Stop();
        testOutputHelper.WriteLine($"Took: {timer.ElapsedMilliseconds.ToString()}ms");
        Assert.Equal(300000, fileContents.Count);
    }
}

[Fact]
public void CanReadCommitParallel()
{
    var timer = new Stopwatch();
    var fileContents = new ConcurrentStack<string>();
    var path = SandboxStandardTestRepo();
    using (var repo = new Repository(path))
    {
        var latestCommit = repo.Head.Tip;
        var tree = latestCommit.Tree;
        timer.Start();
        var fileNames = new List<string>() {"1.txt", "README", "new.txt"}.AsEnumerable();
        Parallel.ForEach(fileNames, (fileName) =>
        {
            for (var i = 0; i < 100000; i++)
            {
                fileContents.Push(ReadEntry(fileName, tree));
            }
        });
        timer.Stop();
        testOutputHelper.WriteLine($"Took: {timer.ElapsedMilliseconds.ToString()}ms");
        Assert.Equal(300000, fileContents.Count);
    }
}

private static string ReadEntry(string name, Tree tree)
{
    var treeEntry = tree[name];
    if (treeEntry != null && treeEntry.Target is Blob blob)
    {
        return blob.GetContentText();
    }

    throw new InvalidOperationException($"{name} must be a Blob");
}
  1. Run against version 0.27.0-preview-0119 (commit: 6329bea). On my machine, CanReadCommitParallel takes 1417ms and CanReadCommit takes 2971ms (which is fine)
  2. Checkout the latest version or the latest release
  3. Run the test again. On my machine, CanReadCommitParallel takes 3731ms and CanReadCommit takes 3250ms!
Expected behavior

Reading files in parallel should be faster than reading files sequentially from the git tree.

Actual behavior

It seems that reading files from the tree in parallel (multi-thread) is not faster. I did a git bisect and it seems that this regression was introduced in https://github.com/libgit2/libgit2sharp/commit/21d4f13ac7c739a5526cf088fbd8765d4ad12f57

Version of LibGit2Sharp (release number or SHA1)

Versions after https://github.com/libgit2/libgit2sharp/commit/21d4f13ac7c739a5526cf088fbd8765d4ad12f57

Operating system(s) tested; .NET runtime tested

.NET 6 and .NET 7.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in CommitFixture.cs by adding and running the sequential and Parallel.ForEach reproductions against the versions named in the report. Compare tree and blob-reading behavior around commit 21d4f13ac7c739a5526cf088fbd8765d4ad12f57 on .NET 6 and .NET 7. Done means parallel reads retain the correct count and are faster than sequential reads on a comparable run.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, git
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.