dotnet / dotnet/skills

Reduce generated dashboard history downloaded by plugin installations

Open
#1,091 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 5h
Merged PRs (30d)
81

Description

## Problem

This repository is distributed as an agent plugin source, so many users will not clone it manually. Their plugin installer will fetch the repository automatically using whatever Git strategy or archive transport that installer implements. As a result, contributor documentation alone cannot ensure that installations avoid the repository's generated dashboard history.

The repository contains three append-only generated snapshot branches:

- `dashboard-token-data`
- `dashboard-eval-data`
- `gh-pages`

Current workflow consumers read only the current files/current branch tips, but the branches retain every historical snapshot. A normal Git clone can fetch objects reachable from all remote branches even though it checks out only `main`. An automated plugin installer may therefore transfer and retain generated history that is unrelated to the plugin content being installed.

The installer-specific impact still needs to be measured. Installers may use a full clone, single-branch clone, shallow/partial clone, sparse checkout, or GitHub archive download; these have materially different costs. We should not assume every installation produces the same local size.

## Evidence

GitHub repository metadata reports approximately **139,194 KiB (~136 MiB)**. That reflects a well delta-compressed server-side pack and is not the same as the amount of uncompressed historical content reachable from repository refs.

`git-sizer` reports approximately **10.2 GiB of total uncompressed blob content** across reachable history.

Measurements from a fresh bare clone:

| Generated branch | Raw historical blob content | Blob count | Optimized on-disk representation |
| --- | ---: | ---: | ---: |
| `gh-pages` | ~4,187.8 MiB | 3,026 | ~94.4 MiB |
| `dashboard-eval-data` | ~1,970.6 MiB | 1,966 | ~47.4 MiB |
| `dashboard-token-data` | ~7,582.0 MiB | 624 | ~67.3 MiB |

These branch totals overlap because branches can reach shared objects. They must **not** be added together and presented as unique repository size.

One concrete contributor is `data/token-usage.json`: the analysis found 258 historical `gh-pages` versions, with individual historical blobs reaching roughly 20–40 MiB.

Fetching disconnected generated branches without `--depth=1` can download and retain unnecessary ancestry. Fetch order, automatic maintenance/repacking, retained refs/reflogs, and duplicate or poorly packed local packs can make an ephemeral checkout substantially larger than GitHub's optimized pack. This explains reports of `.git` directories over 2 GiB, but does not imply every clone or plugin installation will reach that size.

## Why plugin installation changes the priority

For manual contributors, we can recommend an efficient command. Plugin users generally do not control the installer's clone flags, and may not even see that a Git operation occurs. We therefore need a repository- or installer-level solution rather than relying exclusively on user guidance.

The preferred installation behavior should fetch only `main` and only the plugin content required for installation. Depending on installer capabilities, suitable approaches include:

- a single-branch partial clone (`--filter=blob:none --single-branch`)
- a shallow single-branch clone when history is unnecessary (`--depth=1 --single-branch`)
- sparse checkout after a partial clone when only one plugin directory is needed
- a GitHub source archive or packaged plugin artifact that does not include Git history

Archive/package installation is the strongest option when the installer does not need Git history at all.

## Work completed so far

### #1089 — shallow generated-branch operations in Actions

PR #1089 updates all seven relevant sites in `.github/workflows/evaluation.yml`:

- four fetches of `dashboard-token-data` and `dashboard-eval-data`, covering both data generation and `deploy-dashboard`
- three clones of `dashboard-token-data`, `dashboard-eval-data`, and `gh-pages` before publishing

Each operation now uses `--depth=1`. This is behavior-preserving because consumers only check out/show/copy tip snapshot files, and publishers create a normal fast-forward child commit from the shallow-cloned current tip.

The PR does not force-push, rewrite remote history, alter retention, or stop future branch growth. It safely reduces Actions runner transfer and local object retention. `actionlint` v1.7.7 passed.

### #1090 — efficient clone guidance for contributors

PR #1090 documents:

```bash
git clone --filter=blob:none --single-branch https://github.com/dotnet/skills.git
```

for contributors who need `main` history, and:

```bash
git clone --depth=1 --single-branch https://github.com/dotnet/skills.git
```

for CI/read-only use that needs only the latest snapshot. It explains that the shallow option omits older `main` history.

This helps manual users, but it does not control automated agent plugin installers and is therefore only one part of the solution.

## Proposed next steps

### 1. Measure actual plugin installer behavior

For each supported agent/plugin installation path:

- identify whether it uses Git clone, GitHub archives, or packaged artifacts
- record clone/fetch flags and refs requested
- measure network transfer, resulting checkout size, and `.git` size from a clean installation
- verify whether generated dashboard branches are downloaded
- determine whether the installer can use single-branch, shallow, partial, sparse, or archive-based installation without breaking updates

### 2. Optimize automated installation

Where this repository or its marketplace metadata controls installation behavior, prefer an archive/package containing only required plugin files. Otherwise, update the relevant installer to avoid fetching generated branches and history.

The result should not depend on users manually applying clone flags.

### 3. Perform a one-time generated-branch compaction

After #1089 is merged, consider a separately approved maintenance operation:

1. Pause dashboard publishing workflows.
2. Record all three generated-branch tips and create an external backup bundle.
3. Create a parentless commit for each generated branch whose tree exactly matches its current tip.
4. Verify old and new tip trees are identical.
5. Update each branch using `--force-with-lease` pinned to the recorded old SHA.
6. Validate dashboard data, Pages output, and the next normal fast-forward publishing run.
7. Resume workflows.
8. Allow GitHub's server-side unreachable-object cleanup to run; contact GitHub Support if old unreachable objects remain material to repository size.

Replacing each whole generated branch with an equivalent root snapshot is simpler than selectively rewriting it with `git filter-repo`. Existing local clones will not shrink automatically; a fresh clone is safest.

This operation requires explicit maintainer approval because it rewrites the three generated branch histories. It should not rewrite `main`, tags, or unrelated refs.

### 4. Stop renewed growth at the source

One-time compaction resets the generated branches, but append-only publishing will start growing them again. The durable architecture is to move generated dashboard data out of this source repository and deploy Pages without a history-bearing `gh-pages` branch, for example:

- store generated datasets in a dedicated data repository or artifact/object store
- deploy GitHub Pages through Actions upload/deploy artifacts
- remove the generated branches from this repository once migration and rollback validation are complete

Recurring force-push compaction is possible but is not preferred because it introduces workflow races, branch-protection exceptions, and recovery complexity.

## Safety and limitations

- Do not add force pushes to the normal publishing workflow as an incidental optimization.
- Do not compact branches until #1089 is merged and workflow behavior is verified.
- Keep a backup outside this repository; a backup tag in the repository would keep the old objects reachable.
- Use `--force-with-lease`, not an unguarded force push, for an approved one-time rewrite.
- Compaction may not immediately change GitHub's reported size because hidden refs, caches, and server garbage-collection timing can retain objects temporarily.
- Existing clones may retain old tips in refs or reflogs and will not automatically reclaim space.
- Measurements should distinguish raw reachable blob content, optimized pack size, network transfer, working-tree size, and `.git` size.

## Completion criteria

- [ ] #1089 is merged and generated-branch workflow operations remain functional.
- [ ] #1090 is merged so manual contributors have efficient clone guidance.
- [ ] Supported agent/plugin installers are inventoried and measured from clean installs.
- [ ] Automated installation avoids fetching generated dashboard history.
- [ ] Maintainers decide whether to perform the one-time generated-branch compaction.
- [ ] If approved, compaction is backed up, tree-equivalence checked, performed with leases, and validated end to end.
- [ ] Maintainers decide on a durable data/Pages migration that prevents renewed repository-history growth.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/evaluation.yml and the supported agent/plugin installation paths described in the issue. Measure clean installations for transfer, checkout size, .git size, requested refs, and generated-branch access; done means installation avoids unnecessary generated history and maintainers have decided whether to compact or migrate the generated data.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github, github-actions
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.