microsoft / microsoft/VFSForGit

PackfileMaintenanceStep should dynamically resize the batch size

Open
#1,091 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects: performance
Dominant language
C#
Stars
6.1k
Forks
474
Avg merge
2d 4h
Merged PRs (30d)
8

Description

The PackfileMaintenanceStep expires prefetch packs where every object appears in a new pack. It then condenses many packs into one pack using git multi-pack-index repack --batch-size=<size> where "size" is currently "2gb".

We cannot make "2gb" much larger due to issues with unsigned long on Windows. This also limits the amount of work we would do in one step. With this batch size, we anticipate the users with ~100 GB of prefetch data will get to a steady state in 30-60 runs of the step.

Once in a steady state, we anticipate that it will take a long time before the repack command has enough data for a full batch of data.

We can predict if the batch size is too large by examining the pack directory and counting the number of files smaller than 2gb. We could even predict a size that would group all packs into a single pack by adding their sizes and subtracting 1. If this size is larger than 2 * 1024 * 1024 * 1024, then we can stick with the default.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating PackfileMaintenanceStep and its use of git multi-pack-index repack --batch-size=2gb. Read how the pack directory is examined and validate the proposed size calculation, including the Windows unsigned-long limit. Done when the step dynamically selects a batch size for smaller packs, retains the default when the calculated size exceeds 2 * 1024 * 1024 * 1024, and regression coverage verifies the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, git
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.