Move heavy eval fixtures off the skill-consumption critical path
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 81
Description
## Summary
Consumers install skills by registering this repository as a plugin marketplace
(`/plugin marketplace add dotnet/skills`, `codex plugin marketplace add dotnet/skills`).
That operation clones the whole repository. But the shipped product is only
`plugins/**`. The eval fixtures and references under `tests/**` are the majority
of the repository, and every consumer downloads them even though they are never
used at skill-consumption time.
This issue proposes to move the heavy eval fixtures off that critical path.
## Measurements
Taken on branch `abhitejjohn-improve-eval-quality` (draft PR #1118), tracked files only.
| Scope | Files | Uncompressed | Share of repo |
|---|---|---|---|
| Whole repo (tracked) | 2,767 | 15.62 MB | 100% |
| `tests/**` | 2,254 | 9.57 MB | 81.5% files / 61.3% bytes |
| `plugins/**` (shipped skills) | 327 | 2.25 MB | — |
| `eng/**` | 108 | 2.35 MB | — |
- `tests/**` is about **7× the file count** and **4× the bytes** of the actual
product (`plugins/**`).
- Compressed snapshot proxy (`git archive --format=tar.gz HEAD`): full repo
**3.24 MB**, `tests/**` alone **1.13 MB** — the fixtures are about **35% of a
snapshot clone**.
**Largest fixture set** — `tests/dotnet-upgrade`: **999 files / 7.62 MB**, mostly
**954 `.cs`** project fixtures. This one directory is ~80% of all `tests/**` bytes.
Per-plugin fixture bytes (top):
| Directory | Files | MB |
|---|---|---|
| tests/dotnet-upgrade | 999 | 7.62 |
| tests/dotnet-test | 418 | 0.64 |
| tests/dotnet-msbuild | 446 | 0.53 |
| tests/dotnet-diag | 36 | 0.26 |
| tests/dotnet-test-migration | 159 | 0.14 |
**New artifacts in PR #1118 / branch `abhitejjohn-improve-eval-quality`**
(merge-base → branch tip): 326 files changed, +12,194 / −582.
- **305 new files** added, totaling **0.27 MB**.
- **290 of the 305** land in `tests/dotnet-msbuild` (eval fixtures).
The branch is **file-count heavy but byte-light**: it does not add much size, but
it adds many small fixture files. Each new file adds a Git object and a working-tree
entry that every marketplace consumer clones. This is the pattern the repository
repeats each time an eval is strengthened, so the file count grows steadily on the
critical path.
## Why these files are on the critical path
- Skill consumers do not build or test the repo. They only load `plugins/**`.
- The install path (plugin marketplace / `skill-installer`) fetches the repository,
so `tests/**` comes with it.
- Harnesses that clone the repo only to read skills pay the same cost: more objects
to transfer, more inodes to write, slower shallow clones and container image layers.
- Fixture growth is unbounded. Every strengthened eval (like PR #1118) adds more
files that consumers must download but never use.
## Proposed options
### Option A — Separate fixtures repository (for example `dotnet/skills-evals`)
Move `tests/**` fixtures and references to a second repository. This repo keeps
`plugins/**`, `eng/**`, and thin eval specs (`eval.yaml`). CI in the fixtures repo,
or a submodule/checkout step, provides fixtures when evals run.
- **Pros:** removes ~61% of bytes and ~81% of files from the consumer clone;
clean product boundary.
- **Cons:** loses atomic "skill + fixture" changes in one PR; two-repo coordination;
cross-repo versioning needed to keep evals reproducible.
### Option B — Selectively fetched artifact package
Package fixtures as a versioned artifact (NuGet, GitHub Release asset, or a git
archive pinned by tag/SHA). Eval runs download and unpack the pinned version.
- **Pros:** consumers never fetch fixtures; fixtures are versioned and immutable;
works well for CI caching.
- **Cons:** publish step to maintain; offline/air-gapped evals need a local cache
or vendored copy; a new supply-chain surface to secure (checksums, provenance).
### Option C — Sparse-checkout / partial clone guidance
Keep one repo, but document and default consumers to `git clone --filter=blob:none`
or `sparse-checkout` that excludes `tests/**`. Also confirm the marketplace/installer
uses a shallow or sparse fetch.
- **Pros:** no repository split; atomic changes preserved.
- **Cons:** does not shrink history; depends on client support for filtered clone;
the marketplace fetch behavior may not honor it.
### Option D — Git LFS for large fixture blobs
Track large fixture files (for example the `dotnet-upgrade` `.cs` set) with Git LFS.
- **Pros:** working tree stays small unless LFS objects are pulled.
- **Cons:** LFS adds bandwidth quota and tooling requirements; poor fit for many
tiny files (the smart-pointer overhead can exceed the savings); still on the path
unless clients skip LFS.
## Tradeoff summary
| Concern | A (separate repo) | B (artifact package) | C (sparse/partial) | D (LFS) |
|---|---|---|---|---|
| Discoverability | lower | lower | same | same |
| Atomic skill+fixture change | harder | harder | preserved | preserved |
| CI complexity | medium | medium | low | medium |
| Versioning/reproducibility | needs pin | strong (immutable) | same as today | medium |
| Security/supply chain | new repo perms | new publish surface | none new | LFS store |
| Offline evals | needs checkout | needs cache | works | needs LFS pull |
| Consumer clone reduction | highest | highest | high | high |
## Recommendation
Adopt **Option B (selectively fetched, version-pinned fixture artifact)** as the
primary fix, because it fully removes fixtures from the consumer clone while keeping
evals reproducible through an immutable pinned version. Start by relocating the
largest offender, `tests/dotnet-upgrade` (999 files / 7.62 MB), which alone removes
~80% of `tests/**` bytes for a small, well-scoped first step.
As a low-risk immediate mitigation, also apply **Option C**: document and default
consumers to a partial/sparse clone that excludes `tests/**`, and confirm the
marketplace install path fetches shallow or filtered.
Keep thin eval specs (`eval.yaml`) in this repository so evals remain discoverable
and CI can still map a skill to its eval; only the heavy fixtures move.
---
*Filed from an analysis session. Measurements are reproducible with
`git ls-files tests`, `git archive --format=tar.gz HEAD [tests]`, and
`git diff --diff-filter=A abhitejjohn-improve-eval-quality`.*
Contributor guide
Research direction
Start with the consumer entry points (`/plugin marketplace add`, `codex plugin marketplace add`, and `skill-installer`) and compare the shipped `plugins/**` with `tests/dotnet-upgrade` and the thin `eval.yaml` specs. Reproduce the repository-size measurements with `git ls-files tests` and `git archive --format=tar.gz HEAD [tests]`; done means consumers avoid heavy fixtures while evals can fetch a pinned version reproducibly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, git, github
- Domain
- build-system, developer-experience, devops
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100