libgit2 / libgit2/libgit2sharp
Parallel calls to Repository.ObjectDatabase.CreateBlob with identical contents fails.
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 3.5k
- Forks
- 925
- PR merge metrics
- No merged PRs in 30d
Description
I would like to insert blobs from memory into an object database in parallel (based on the technique described at http://stackoverflow.com/a/16244234). This works fine most of the time, but is subject to race conditions if the contents are the same. There are two classes of errors I observe:
1. Exception due to FILE_EXISTS conflict when trying to create the directory with the first two characters of the OID under objects.
2. NULL blob returned from CreateBlob.
Here's some example code to reproduce the problem:
using System;
using System.Text;
using System.IO;
using System.Threading.Tasks;
using LibGit2Sharp;
namespace GitODBStress
{
class Program
{
static void Main()
{
string repoPath = Path.Combine(Path.GetTempPath(), "Git_Parallel_Odb");
int iterationCount = 100;
if (!Directory.Exists(repoPath))
{
Directory.CreateDirectory(repoPath);
}
var path = Repository.Init(repoPath, isBare:true);
Console.WriteLine("Made repo at {0}", path);
var repo = new Repository(path);
var tasks = new Task[2];
for (int i = 0; i < iterationCount; ++i)
{
var iter = i;
byte[] buf = Encoding.ASCII.GetBytes("foo" + iter.ToString());
for (int taskNo = 0; taskNo < 2; ++taskNo)
{
tasks[taskNo] = Task.Factory.StartNew(() =>
{
MemoryStream ms = new MemoryStream(buf);
for (; ; )
{
try
{
var blob = repo.ObjectDatabase.CreateBlob(ms);
if (blob == null)
{
Console.WriteLine("ERROR: NULL blob on iteration {0}", iter);
continue;
}
break;
}
catch (LibGit2SharpException e)
{
Console.WriteLine("ERROR: Caught LibGit2SharpException on iteration {0}: {1}", iter, e.Message);
if (e.Message.Contains("Cannot create a file when that file already exists."))
{
continue;
}
}
}
});
}
Task.WaitAll(tasks);
}
repo.Dispose();
// clean up temp repo files
var files = Directory.GetFiles(repoPath, "*.*", SearchOption.AllDirectories);
foreach (var f in files)
{
var attrib = File.GetAttributes(f);
if (attrib.HasFlag(FileAttributes.ReadOnly))
{
File.SetAttributes(f, attrib & ~FileAttributes.ReadOnly);
}
}
Directory.Delete(repoPath, true);
}
}
}
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Repository.ObjectDatabase.CreateBlob and run the C# reproduction against a bare repository. Trace the parallel path for identical contents and the reported FILE_EXISTS and null-blob outcomes. Done means concurrent calls complete without either failure while creating the expected blob.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, git
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100