rust-lang / rust-lang/git2-rs

git2 holds open deleted packfiles, effectively leaking disk space

Open
#904 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.1k
Forks
450
Avg merge
11m
Merged PRs (30d)
1

Description

We have a long-running daemon that does some stuff with a repository.

We noticed that the disk backing the repository was "full":

thing@thing-0:/var/cache/thing$ du -chs .
7.3G	.
7.3G	total
thing@thing-0:/var/cache/thing$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         16G   16G     0 100% /var/cache/thing

That might see illogical, but that's a hint that deleted files are being held open. If we ls our daemon's open files, that confirms it:

0 -> /dev/null
1 -> 'pipe:[16731674]'
10 -> 'socket:[16737720]'
11 -> 'anon_inode:[eventpoll]'
12 -> 'anon_inode:[eventfd]'
13 -> 'anon_inode:[eventpoll]'
14 -> 'anon_inode:[eventfd]'
15 -> 'anon_inode:[eventpoll]'
16 -> 'socket:[16729776]'
17 -> 'anon_inode:[eventpoll]'
18 -> 'anon_inode:[eventfd]'
19 -> 'anon_inode:[eventpoll]'
2 -> 'pipe:[16731675]'
20 -> 'socket:[16729776]'
21 -> '/var/cache/thing/monorepo/.git/objects/pack/pack-a02eda73e77509f55e734b4f80f0e49c60093184.pack (deleted)'
22 -> '/var/cache/thing/environments/.git/objects/pack/pack-be204a28036cbc4cd01c2248a6ee2168a1989a96.pack (deleted)'
23 -> '/var/cache/thing/environments/.git/objects/pack/pack-6e09f3d300e72f084b05ba7bc1ccce40c10f582c.pack (deleted)'
24 -> '/var/cache/thing/monorepo/.git/objects/pack/pack-daec94cd70f17f2b1d3a638ce87cb4c086ce4629.pack (deleted)'
25 -> '/var/cache/thing/environments/.git/objects/pack/pack-2ad6f4562cd68fbf4ecc4d5af17579aa590e7263.pack (deleted)'
26 -> '/var/cache/thing/environments/.git/objects/pack/pack-aed3c72e24e8ec9a5db6b11bc2a81f071b5596f6.pack (deleted)'
27 -> '/var/cache/thing/environments/.git/objects/pack/pack-d4e7ed286a0e2e79cd36688ac33c5f98948e6b6a.pack (deleted)'
28 -> '/var/cache/thing/environments/.git/objects/pack/pack-6ffb344d987435d1cca7934ed89e7ddf9d6d79a2.pack (deleted)'
29 -> '/var/cache/thing/environments/.git/objects/pack/pack-09e7c333cc2b2dbb5b30550298b31f30ea4439c7.pack (deleted)'
3 -> 'anon_inode:[eventpoll]'

So, we can see here that we're holding a bunch of packfiles, which explains why the used space is so much higher than the du total.

We keep the git2::Repository object around: I think I didn't know if it was expensive to create, and we only have two repos, so we just create the git2::Repository objects are startup, and then keep them as global state.

But that seems to cause a "disk" leak, over repeated fetches & repacks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the long-running daemon scenario with persistent git2::Repository objects and repeated fetches or repacks. Inspect the repository's open files for deleted packfiles and trace repository lifetime and packfile handling; done means repeated operations no longer retain deleted packfiles or exhaust disk space.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, rust
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.