dotnet / dotnet/skills

Move heavy eval fixtures off the skill-consumption critical path

Open
#1,121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 5h
Merged PRs (30d)
81

Description

## Summary

Consumers install skills by registering this repository as a plugin marketplace
(`/plugin marketplace add dotnet/skills`, `codex plugin marketplace add dotnet/skills`).
That operation clones the whole repository. But the shipped product is only
`plugins/**`. The eval fixtures and references under `tests/**` are the majority
of the repository, and every consumer downloads them even though they are never
used at skill-consumption time.

This issue proposes to move the heavy eval fixtures off that critical path.

## Measurements

Taken on branch `abhitejjohn-improve-eval-quality` (draft PR #1118), tracked files only.

| Scope | Files | Uncompressed | Share of repo |
|---|---|---|---|
| Whole repo (tracked) | 2,767 | 15.62 MB | 100% |
| `tests/**` | 2,254 | 9.57 MB | 81.5% files / 61.3% bytes |
| `plugins/**` (shipped skills) | 327 | 2.25 MB | — |
| `eng/**` | 108 | 2.35 MB | — |

- `tests/**` is about **7× the file count** and **4× the bytes** of the actual
product (`plugins/**`).
- Compressed snapshot proxy (`git archive --format=tar.gz HEAD`): full repo
**3.24 MB**, `tests/**` alone **1.13 MB** — the fixtures are about **35% of a
snapshot clone**.

**Largest fixture set** — `tests/dotnet-upgrade`: **999 files / 7.62 MB**, mostly
**954 `.cs`** project fixtures. This one directory is ~80% of all `tests/**` bytes.

Per-plugin fixture bytes (top):

| Directory | Files | MB |
|---|---|---|
| tests/dotnet-upgrade | 999 | 7.62 |
| tests/dotnet-test | 418 | 0.64 |
| tests/dotnet-msbuild | 446 | 0.53 |
| tests/dotnet-diag | 36 | 0.26 |
| tests/dotnet-test-migration | 159 | 0.14 |

**New artifacts in PR #1118 / branch `abhitejjohn-improve-eval-quality`**
(merge-base → branch tip): 326 files changed, +12,194 / −582.
- **305 new files** added, totaling **0.27 MB**.
- **290 of the 305** land in `tests/dotnet-msbuild` (eval fixtures).

The branch is **file-count heavy but byte-light**: it does not add much size, but
it adds many small fixture files. Each new file adds a Git object and a working-tree
entry that every marketplace consumer clones. This is the pattern the repository
repeats each time an eval is strengthened, so the file count grows steadily on the
critical path.

## Why these files are on the critical path

- Skill consumers do not build or test the repo. They only load `plugins/**`.
- The install path (plugin marketplace / `skill-installer`) fetches the repository,
so `tests/**` comes with it.
- Harnesses that clone the repo only to read skills pay the same cost: more objects
to transfer, more inodes to write, slower shallow clones and container image layers.
- Fixture growth is unbounded. Every strengthened eval (like PR #1118) adds more
files that consumers must download but never use.

## Proposed options

### Option A — Separate fixtures repository (for example `dotnet/skills-evals`)
Move `tests/**` fixtures and references to a second repository. This repo keeps
`plugins/**`, `eng/**`, and thin eval specs (`eval.yaml`). CI in the fixtures repo,
or a submodule/checkout step, provides fixtures when evals run.
- **Pros:** removes ~61% of bytes and ~81% of files from the consumer clone;
clean product boundary.
- **Cons:** loses atomic "skill + fixture" changes in one PR; two-repo coordination;
cross-repo versioning needed to keep evals reproducible.

### Option B — Selectively fetched artifact package
Package fixtures as a versioned artifact (NuGet, GitHub Release asset, or a git
archive pinned by tag/SHA). Eval runs download and unpack the pinned version.
- **Pros:** consumers never fetch fixtures; fixtures are versioned and immutable;
works well for CI caching.
- **Cons:** publish step to maintain; offline/air-gapped evals need a local cache
or vendored copy; a new supply-chain surface to secure (checksums, provenance).

### Option C — Sparse-checkout / partial clone guidance
Keep one repo, but document and default consumers to `git clone --filter=blob:none`
or `sparse-checkout` that excludes `tests/**`. Also confirm the marketplace/installer
uses a shallow or sparse fetch.
- **Pros:** no repository split; atomic changes preserved.
- **Cons:** does not shrink history; depends on client support for filtered clone;
the marketplace fetch behavior may not honor it.

### Option D — Git LFS for large fixture blobs
Track large fixture files (for example the `dotnet-upgrade` `.cs` set) with Git LFS.
- **Pros:** working tree stays small unless LFS objects are pulled.
- **Cons:** LFS adds bandwidth quota and tooling requirements; poor fit for many
tiny files (the smart-pointer overhead can exceed the savings); still on the path
unless clients skip LFS.

## Tradeoff summary

| Concern | A (separate repo) | B (artifact package) | C (sparse/partial) | D (LFS) |
|---|---|---|---|---|
| Discoverability | lower | lower | same | same |
| Atomic skill+fixture change | harder | harder | preserved | preserved |
| CI complexity | medium | medium | low | medium |
| Versioning/reproducibility | needs pin | strong (immutable) | same as today | medium |
| Security/supply chain | new repo perms | new publish surface | none new | LFS store |
| Offline evals | needs checkout | needs cache | works | needs LFS pull |
| Consumer clone reduction | highest | highest | high | high |

## Recommendation

Adopt **Option B (selectively fetched, version-pinned fixture artifact)** as the
primary fix, because it fully removes fixtures from the consumer clone while keeping
evals reproducible through an immutable pinned version. Start by relocating the
largest offender, `tests/dotnet-upgrade` (999 files / 7.62 MB), which alone removes
~80% of `tests/**` bytes for a small, well-scoped first step.

As a low-risk immediate mitigation, also apply **Option C**: document and default
consumers to a partial/sparse clone that excludes `tests/**`, and confirm the
marketplace install path fetches shallow or filtered.

Keep thin eval specs (`eval.yaml`) in this repository so evals remain discoverable
and CI can still map a skill to its eval; only the heavy fixtures move.

---

*Filed from an analysis session. Measurements are reproducible with
`git ls-files tests`, `git archive --format=tar.gz HEAD [tests]`, and
`git diff --diff-filter=A abhitejjohn-improve-eval-quality`.*

Contributor guide

Open the contributing guide

Research direction

Start with the consumer entry points (`/plugin marketplace add`, `codex plugin marketplace add`, and `skill-installer`) and compare the shipped `plugins/**` with `tests/dotnet-upgrade` and the thin `eval.yaml` specs. Reproduce the repository-size measurements with `git ls-files tests` and `git archive --format=tar.gz HEAD [tests]`; done means consumers avoid heavy fixtures while evals can fetch a pinned version reproducibly.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, git, github
Domain
build-system, developer-experience, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.