microsoft / microsoft/hve-core
feat: Add post-build `.vsix` content validator (`Validate-ExtensionPackage.ps1`)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
### Issue Description
Create a post-build validator that unpacks `.vsix` extension packages and verifies their contents match the `contributes` declarations in `package.json`. This closes the gap between source-level collection validation (`Validate-Collections.ps1`) and actual packaged output, as mentioned in #436.
### Motivation
`vsce package` is a zipper, not a validator. It follows `.vscodeignore` rules to determine which files to include and writes them into a `.vsix` archive (ZIP format with a manifest). It does **not** verify that files referenced in `contributes` entries actually exist in the final package.
In a multi-collection build where `Prepare-Extension.ps1` dynamically filters artifacts per collection, a broken filter can produce a `.vsix` that:
* Packages successfully (exit code 0)
* Installs in VS Code without error
* Silently fails at runtime because referenced agents, prompts, instructions, or skills are missing from the archive
This class of error is invisible to the current pipeline. Source-level validation (`Validate-Collections.ps1`) only checks that collection manifests reference files that exist on disk — it does not verify the build output.
### Deliverables
#### 1. `Validate-ExtensionPackage.ps1` Script
Create `scripts/extension/Validate-ExtensionPackage.ps1` that accepts a `.vsix` file path and validates its contents.
**Contributes validation** — cross-reference these `contributes` sections against actual files in the archive:
| Contributes Key | Artifact Type | Path Property |
|-----------------|---------------|---------------|
| `chatAgents` | Agents and subagents (`.agent.md`) | `path` |
| `chatPromptFiles` | Prompts (`.prompt.md`) | `path` |
| `chatInstructions` | Instructions (`.instructions.md`) | `path` |
| `chatSkills` | Skills (`SKILL.md`) | `path` |
For each entry in each `contributes` array, verify the referenced `path` value resolves to an actual file inside the `.vsix` archive.
**Structural validation:**
| Check | Description |
|-------|-------------|
| `package.json` parseable | Confirm valid JSON with required fields (`name`, `version`, `publisher`, `engines.vscode`) |
| Version consistency | Internal `package.json` version matches expected version (when provided via parameter) |
| Icon file present | `icon` field in `package.json` resolves to an actual file in the archive |
| LICENSE file present | Archive contains a LICENSE file at the expected location |
| Size threshold | Warn if total archive size or any single file exceeds configurable thresholds (catches accidental inclusion of `node_modules`, logs, etc.) |
**Script conventions** (following existing patterns from `Validate-Marketplace.ps1` and `Validate-Collections.ps1`):
* Pure test functions: `Test-VsixContributes`, `Test-VsixStructure`, `Test-VsixManifest`
* I/O functions: `Expand-VsixToTemp` (extract to temp directory, clean up in `finally` block)
* Orchestration function: `Invoke-ExtensionPackageValidation`
* Return hashtable with `Success`, `Errors` array, `ErrorCount`
* Import and use `CIHelpers.psm1` for `Write-CIAnnotation` and `Set-CIOutput`
* Support comment-based help (`.SYNOPSIS`, `.DESCRIPTION`, `.PARAMETER`, `.EXAMPLE`)
* `#Requires -Version 7.0`
* Copyright header: `# Copyright (c) Microsoft Corporation.` / `# SPDX-License-Identifier: MIT`
#### 2. CI Workflow Integration (Primary)
Update `.github/workflows/extension-package.yml` to run the validator after each `vsce package` step, before the `.vsix` artifact is uploaded. The validator runs per collection in the matrix strategy, validating the `.vsix` that was built for that collection. This is the primary integration point, every collection build in CI is validated automatically.
The workflow already produces `vsix-file` and `version` as step outputs from `Package-Extension.ps1`. Pass these to the validator:
```yaml
- name: Validate extension package
shell: pwsh
run: |
scripts/extension/Validate-ExtensionPackage.ps1 `
-VsixPath "extension/${{ steps.package.outputs.vsix-file }}" `
-ExpectedVersion "${{ steps.package.outputs.version }}"
```
#### 3. npm Script Registration
Add an npm script to `package.json`:
```json
"validate:extension-package": "pwsh -File scripts/extension/Validate-ExtensionPackage.ps1"
```
This is **not** added to `lint:all` because it requires a built `.vsix` as input (not a source-level check). The script is available for local testing after a local build:
```bash
# Build a collection package, then validate it
pwsh ./scripts/extension/Prepare-Extension.ps1 -Collection collections/hve-core.collection.yml
pwsh ./scripts/extension/Package-Extension.ps1 -Collection collections/hve-core.collection.yml
# Validate the built .vsix (pass the path to the generated file)
npm run validate:extension-package -- -VsixPath "extension/hve-core-3.1.46.vsix"
# Or validate with version check
npm run validate:extension-package -- -VsixPath "extension/hve-core-3.1.46.vsix" -ExpectedVersion "3.1.46"
```
#### 4. Documentation Updates
Update `extension/PACKAGING.md` in the "Testing Collection Builds Locally" section to add a post-build validation step after the existing packaging commands. The extension packaging validator is not part of the general contributor workflow for adding artifacts, it is a packaging and local testing tool for verifying built `.vsix` output.
### Acceptance Criteria
* [ ] `Validate-ExtensionPackage.ps1` unpacks a `.vsix` and validates `contributes` entries (`chatAgents`, `chatPromptFiles`, `chatInstructions`, `chatSkills`) against actual archive contents
* [ ] Structural checks pass: `package.json` parseable, icon present, LICENSE present, version consistent
* [ ] Size threshold warnings for oversized archives or individual files
* [ ] CI workflow calls the validator after `vsce package` for each collection in the matrix (primary integration)
* [ ] npm script `validate:extension-package` registered in `package.json` for local testing
* [ ] `extension/PACKAGING.md` updated with post-build validation in the "Testing Collection Builds Locally" section
* [ ] Script follows existing conventions (pure functions, `CIHelpers.psm1`, comment-based help, hashtable return)
### Technical Notes
* A `.vsix` is a ZIP archive. Use `Expand-Archive` to extract to a temp directory, then inspect contents. Clean up in a `finally` block.
* The `extension/` prefix in contributes paths maps to the archive root (minus the `extension/` prefix that `vsce` strips during packaging, verify actual path structure inside the `.vsix`).
* The validator should exit with a non-zero code when errors are found, allowing CI to gate on validation failures.
* Consider a `-ExpectedVersion` parameter for version consistency checks, populated from the packaging step's output.
* The `contributes` arrays use `{ "path": "..." }` objects. The `path` values are relative to the extension root inside the archive.
### Related
* #436 - related issue (Publishing Infrastructure and CI/CD)
* `scripts/collections/Validate-Collections.ps1` - Source-level collection validation (pre-build)
* `scripts/plugins/Validate-Marketplace.ps1` - CLI plugin marketplace validation (pattern reference)
* `scripts/extension/Package-Extension.ps1` - Packaging orchestration (produces the `.vsix`)
* `scripts/extension/Prepare-Extension.ps1` - Artifact discovery and `contributes` generation
### Additional Context
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/extension/Package-Extension.ps1 and Prepare-Extension.ps1, then compare validation patterns in scripts/collections/Validate-Collections.ps1 and scripts/plugins/Validate-Marketplace.ps1. Read .github/workflows/extension-package.yml for the packaging step outputs and extension/PACKAGING.md for the local workflow. Done means the validator, CI step, npm registration, and documentation update cover every listed acceptance criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, powershell
- Domain
- build-system, ci-cd, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100