Memory corruption in index handling
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 5.8k
- Forks
- 704
- PR merge metrics
- No merged PRs in 30d
Description
The following setup pretty reliably produces error messages that are indicative of memory corruption somewhere in Index or IndexEntry, I suspect related to a use-after-free of Index. Save the following file as test.mjs:
import NodeGit from "nodegit";
async function overwriteIndexFromFile(index, path) {
const newIndex = await NodeGit.Index.open(path);
await index.removeAll();
for (const e of newIndex.entries()) {
await index.add(e);
}
}
const path = "testrepo/.git/index";
const index = await NodeGit.Index.open(path);
for (var i = 0; i < 100; i++) {
await overwriteIndexFromFile(index, path);
}
and then set up a test repo with a fairly large index (it is possible, but much rarer, to hit it with a smaller index):
$ git init testrepo
$ cd testrepo
$ for i in {1..20000}; do touch file"$i"; done
$ git add .
$ git commit -m "Initial commit"
$ cd ..
$ yarn install nodegit
$ node test.mjs
node:internal/process/esm_loader:94
internalBinding('errors').triggerUncaughtException(
^
[Error: invalid path: ''] { errno: -1, errorFunction: 'Index.add' }
This is a minimized version of the overwriteIndexFromFile function in twosigma/git-meta. See twosigma/git-meta#882 for a few more of my notes if you're curious - notably, leaking newIndex (by appending it to a global array) appears to avoid the problem, as does running GC in some spots. I think the overwriteIndexFromFile function is validly written and doesn't do anything unusual, but If the answer is "you're holding it wrong," that's still helpful (though I'd still claim that it shouldn't be possible to cause memory corruption by holding it wrong :) ).
The specific errors vary depending on your luck, at least in the un-minimized version: it's usually an invalid path or an invalid mode. If you are careful to log the IndexEntry's contents you will notice other corruption in the filename (@ signs and other weirdness clobbering bytes from the original filename) that doesn't actually trigger a validation error in libgit2 and presumably goes ahead and constructs an erroneous index. And exactly one time, I got a segfault in strlen as called from v8::String::NewFromUtf8.
If there's anything I can provide or try out to be helpful for debugging this, let me know!
System information
- node version: v16.14.2 (also reproduced on v18.9.0; we have not seen this on 8.x, but it may just be luck)
- npm or yarn version: yarn 1.22.18 (also reproduced on 1.23.2)
- OS/version/architecture: Debian 11 x86_64 (also reproduced on Debian 10 x86_64)
- Applicable nodegit version: 0.27.0 (also reproduced on 0.25.1, 0.28.0-alpha.3, 0.28.0-alpha.4, and 0.28.0-alpha.20; haven't tried HEAD yet)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the provided test.mjs script and its repeated Index.open, removeAll, and add calls against a large repository index. Trace the Index and IndexEntry handling involved in those entry points, then verify that repeated execution no longer produces invalid paths, corrupted filenames, or a segfault.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100