microsoft / microsoft/hve-core

feat: Add post-build `.vsix` content validator (`Validate-ExtensionPackage.ps1`)

Open
#963 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension feature infrastructure
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

### Issue Description

Create a post-build validator that unpacks `.vsix` extension packages and verifies their contents match the `contributes` declarations in `package.json`. This closes the gap between source-level collection validation (`Validate-Collections.ps1`) and actual packaged output, as mentioned in #436.

### Motivation

`vsce package` is a zipper, not a validator. It follows `.vscodeignore` rules to determine which files to include and writes them into a `.vsix` archive (ZIP format with a manifest). It does **not** verify that files referenced in `contributes` entries actually exist in the final package.

In a multi-collection build where `Prepare-Extension.ps1` dynamically filters artifacts per collection, a broken filter can produce a `.vsix` that:

* Packages successfully (exit code 0)
* Installs in VS Code without error
* Silently fails at runtime because referenced agents, prompts, instructions, or skills are missing from the archive

This class of error is invisible to the current pipeline. Source-level validation (`Validate-Collections.ps1`) only checks that collection manifests reference files that exist on disk — it does not verify the build output.

### Deliverables

#### 1. `Validate-ExtensionPackage.ps1` Script

Create `scripts/extension/Validate-ExtensionPackage.ps1` that accepts a `.vsix` file path and validates its contents.

**Contributes validation** — cross-reference these `contributes` sections against actual files in the archive:

| Contributes Key | Artifact Type | Path Property |
|-----------------|---------------|---------------|
| `chatAgents` | Agents and subagents (`.agent.md`) | `path` |
| `chatPromptFiles` | Prompts (`.prompt.md`) | `path` |
| `chatInstructions` | Instructions (`.instructions.md`) | `path` |
| `chatSkills` | Skills (`SKILL.md`) | `path` |

For each entry in each `contributes` array, verify the referenced `path` value resolves to an actual file inside the `.vsix` archive.

**Structural validation:**

| Check | Description |
|-------|-------------|
| `package.json` parseable | Confirm valid JSON with required fields (`name`, `version`, `publisher`, `engines.vscode`) |
| Version consistency | Internal `package.json` version matches expected version (when provided via parameter) |
| Icon file present | `icon` field in `package.json` resolves to an actual file in the archive |
| LICENSE file present | Archive contains a LICENSE file at the expected location |
| Size threshold | Warn if total archive size or any single file exceeds configurable thresholds (catches accidental inclusion of `node_modules`, logs, etc.) |

**Script conventions** (following existing patterns from `Validate-Marketplace.ps1` and `Validate-Collections.ps1`):

* Pure test functions: `Test-VsixContributes`, `Test-VsixStructure`, `Test-VsixManifest`
* I/O functions: `Expand-VsixToTemp` (extract to temp directory, clean up in `finally` block)
* Orchestration function: `Invoke-ExtensionPackageValidation`
* Return hashtable with `Success`, `Errors` array, `ErrorCount`
* Import and use `CIHelpers.psm1` for `Write-CIAnnotation` and `Set-CIOutput`
* Support comment-based help (`.SYNOPSIS`, `.DESCRIPTION`, `.PARAMETER`, `.EXAMPLE`)
* `#Requires -Version 7.0`
* Copyright header: `# Copyright (c) Microsoft Corporation.` / `# SPDX-License-Identifier: MIT`

#### 2. CI Workflow Integration (Primary)

Update `.github/workflows/extension-package.yml` to run the validator after each `vsce package` step, before the `.vsix` artifact is uploaded. The validator runs per collection in the matrix strategy, validating the `.vsix` that was built for that collection. This is the primary integration point, every collection build in CI is validated automatically.

The workflow already produces `vsix-file` and `version` as step outputs from `Package-Extension.ps1`. Pass these to the validator:

```yaml
- name: Validate extension package
shell: pwsh
run: |
scripts/extension/Validate-ExtensionPackage.ps1 `
-VsixPath "extension/${{ steps.package.outputs.vsix-file }}" `
-ExpectedVersion "${{ steps.package.outputs.version }}"
```

#### 3. npm Script Registration

Add an npm script to `package.json`:

```json
"validate:extension-package": "pwsh -File scripts/extension/Validate-ExtensionPackage.ps1"
```

This is **not** added to `lint:all` because it requires a built `.vsix` as input (not a source-level check). The script is available for local testing after a local build:

```bash
# Build a collection package, then validate it
pwsh ./scripts/extension/Prepare-Extension.ps1 -Collection collections/hve-core.collection.yml
pwsh ./scripts/extension/Package-Extension.ps1 -Collection collections/hve-core.collection.yml

# Validate the built .vsix (pass the path to the generated file)
npm run validate:extension-package -- -VsixPath "extension/hve-core-3.1.46.vsix"

# Or validate with version check
npm run validate:extension-package -- -VsixPath "extension/hve-core-3.1.46.vsix" -ExpectedVersion "3.1.46"
```

#### 4. Documentation Updates

Update `extension/PACKAGING.md` in the "Testing Collection Builds Locally" section to add a post-build validation step after the existing packaging commands. The extension packaging validator is not part of the general contributor workflow for adding artifacts, it is a packaging and local testing tool for verifying built `.vsix` output.

### Acceptance Criteria

* [ ] `Validate-ExtensionPackage.ps1` unpacks a `.vsix` and validates `contributes` entries (`chatAgents`, `chatPromptFiles`, `chatInstructions`, `chatSkills`) against actual archive contents
* [ ] Structural checks pass: `package.json` parseable, icon present, LICENSE present, version consistent
* [ ] Size threshold warnings for oversized archives or individual files
* [ ] CI workflow calls the validator after `vsce package` for each collection in the matrix (primary integration)
* [ ] npm script `validate:extension-package` registered in `package.json` for local testing
* [ ] `extension/PACKAGING.md` updated with post-build validation in the "Testing Collection Builds Locally" section
* [ ] Script follows existing conventions (pure functions, `CIHelpers.psm1`, comment-based help, hashtable return)

### Technical Notes

* A `.vsix` is a ZIP archive. Use `Expand-Archive` to extract to a temp directory, then inspect contents. Clean up in a `finally` block.
* The `extension/` prefix in contributes paths maps to the archive root (minus the `extension/` prefix that `vsce` strips during packaging, verify actual path structure inside the `.vsix`).
* The validator should exit with a non-zero code when errors are found, allowing CI to gate on validation failures.
* Consider a `-ExpectedVersion` parameter for version consistency checks, populated from the packaging step's output.
* The `contributes` arrays use `{ "path": "..." }` objects. The `path` values are relative to the extension root inside the archive.

### Related

* #436 - related issue (Publishing Infrastructure and CI/CD)
* `scripts/collections/Validate-Collections.ps1` - Source-level collection validation (pre-build)
* `scripts/plugins/Validate-Marketplace.ps1` - CLI plugin marketplace validation (pattern reference)
* `scripts/extension/Package-Extension.ps1` - Packaging orchestration (produces the `.vsix`)
* `scripts/extension/Prepare-Extension.ps1` - Artifact discovery and `contributes` generation

### Additional Context

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/extension/Package-Extension.ps1 and Prepare-Extension.ps1, then compare validation patterns in scripts/collections/Validate-Collections.ps1 and scripts/plugins/Validate-Marketplace.ps1. Read .github/workflows/extension-package.yml for the packaging step outputs and extension/PACKAGING.md for the local workflow. Done means the validator, CI step, npm registration, and documentation update cover every listed acceptance criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, powershell
Domain
build-system, ci-cd, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.