libgit2 / libgit2/libgit2sharp

Performance regression when reading files from the tree in parallel

未關閉
#2,076 6 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

主要語言
C#
星號
3.5k
分支
925
PR 合併指標
30 天內沒有已合併 PR

描述

Reproduction steps
  1. Add the following test cases in CommitFixture.cs:
[Fact]
public void CanReadCommit()
{
    var timer = new Stopwatch();
    var fileContents = new ConcurrentStack<string>();
    var path = SandboxStandardTestRepo();
    using (var repo = new Repository(path))
    {
        var latestCommit = repo.Head.Tip;
        var tree = latestCommit.Tree;
        timer.Start();
        for (var i = 0; i < 100000; i++)
        {
            fileContents.Push(ReadEntry("1.txt", tree));
            fileContents.Push(ReadEntry("README", tree));
            fileContents.Push(ReadEntry("new.txt", tree));
        }
        timer.Stop();
        testOutputHelper.WriteLine($"Took: {timer.ElapsedMilliseconds.ToString()}ms");
        Assert.Equal(300000, fileContents.Count);
    }
}

[Fact]
public void CanReadCommitParallel()
{
    var timer = new Stopwatch();
    var fileContents = new ConcurrentStack<string>();
    var path = SandboxStandardTestRepo();
    using (var repo = new Repository(path))
    {
        var latestCommit = repo.Head.Tip;
        var tree = latestCommit.Tree;
        timer.Start();
        var fileNames = new List<string>() {"1.txt", "README", "new.txt"}.AsEnumerable();
        Parallel.ForEach(fileNames, (fileName) =>
        {
            for (var i = 0; i < 100000; i++)
            {
                fileContents.Push(ReadEntry(fileName, tree));
            }
        });
        timer.Stop();
        testOutputHelper.WriteLine($"Took: {timer.ElapsedMilliseconds.ToString()}ms");
        Assert.Equal(300000, fileContents.Count);
    }
}

private static string ReadEntry(string name, Tree tree)
{
    var treeEntry = tree[name];
    if (treeEntry != null && treeEntry.Target is Blob blob)
    {
        return blob.GetContentText();
    }

    throw new InvalidOperationException($"{name} must be a Blob");
}
  1. Run against version 0.27.0-preview-0119 (commit: 6329bea). On my machine, CanReadCommitParallel takes 1417ms and CanReadCommit takes 2971ms (which is fine)
  2. Checkout the latest version or the latest release
  3. Run the test again. On my machine, CanReadCommitParallel takes 3731ms and CanReadCommit takes 3250ms!
Expected behavior

Reading files in parallel should be faster than reading files sequentially from the git tree.

Actual behavior

It seems that reading files from the tree in parallel (multi-thread) is not faster. I did a git bisect and it seems that this regression was introduced in https://github.com/libgit2/libgit2sharp/commit/21d4f13ac7c739a5526cf088fbd8765d4ad12f57

Version of LibGit2Sharp (release number or SHA1)

Versions after https://github.com/libgit2/libgit2sharp/commit/21d4f13ac7c739a5526cf088fbd8765d4ad12f57

Operating system(s) tested; .NET runtime tested

.NET 6 and .NET 7.

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

從 CommitFixture.cs 開始,針對報告中提到的版本新增並執行循序執行和 Parallel.ForEach 重現。比較 .NET 6 和 .NET 7 上 commit 21d4f13ac7c739a5526cf088fbd8765d4ad12f57 附近的 tree 和 blob 讀取行為。當平行讀取維持正確的計數,並且在可比較的執行中比循序讀取更快時,即表示完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
csharp, git
領域
performance
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
42/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。