nodegit / nodegit/nodegit

Memory corruption in index handling

Open
#1,970 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
5.8k
Forks
704
PR merge metrics
No merged PRs in 30d

Description

The following setup pretty reliably produces error messages that are indicative of memory corruption somewhere in Index or IndexEntry, I suspect related to a use-after-free of Index. Save the following file as test.mjs:

import NodeGit from "nodegit";

async function overwriteIndexFromFile(index, path) {
    const newIndex = await NodeGit.Index.open(path);
    await index.removeAll();
    for (const e of newIndex.entries()) {
        await index.add(e);
    }
}

const path = "testrepo/.git/index";
const index = await NodeGit.Index.open(path);
for (var i = 0; i < 100; i++) {
    await overwriteIndexFromFile(index, path);
}

and then set up a test repo with a fairly large index (it is possible, but much rarer, to hit it with a smaller index):

$ git init testrepo
$ cd testrepo
$ for i in {1..20000}; do touch file"$i"; done
$ git add .
$ git commit -m "Initial commit"
$ cd ..
$ yarn install nodegit
$ node test.mjs
node:internal/process/esm_loader:94
    internalBinding('errors').triggerUncaughtException(
                              ^

[Error: invalid path: ''] { errno: -1, errorFunction: 'Index.add' }

This is a minimized version of the overwriteIndexFromFile function in twosigma/git-meta. See twosigma/git-meta#882 for a few more of my notes if you're curious - notably, leaking newIndex (by appending it to a global array) appears to avoid the problem, as does running GC in some spots. I think the overwriteIndexFromFile function is validly written and doesn't do anything unusual, but If the answer is "you're holding it wrong," that's still helpful (though I'd still claim that it shouldn't be possible to cause memory corruption by holding it wrong :) ).

The specific errors vary depending on your luck, at least in the un-minimized version: it's usually an invalid path or an invalid mode. If you are careful to log the IndexEntry's contents you will notice other corruption in the filename (@ signs and other weirdness clobbering bytes from the original filename) that doesn't actually trigger a validation error in libgit2 and presumably goes ahead and constructs an erroneous index. And exactly one time, I got a segfault in strlen as called from v8::String::NewFromUtf8.

If there's anything I can provide or try out to be helpful for debugging this, let me know!

System information

  • node version: v16.14.2 (also reproduced on v18.9.0; we have not seen this on 8.x, but it may just be luck)
  • npm or yarn version: yarn 1.22.18 (also reproduced on 1.23.2)
  • OS/version/architecture: Debian 11 x86_64 (also reproduced on Debian 10 x86_64)
  • Applicable nodegit version: 0.27.0 (also reproduced on 0.25.1, 0.28.0-alpha.3, 0.28.0-alpha.4, and 0.28.0-alpha.20; haven't tried HEAD yet)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the provided test.mjs script and its repeated Index.open, removeAll, and add calls against a large repository index. Trace the Index and IndexEntry handling involved in those entry points, then verify that repeated execution no longer produces invalid paths, corrupted filenames, or a segfault.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.