Add consumer-install e2e test: pack + install + run docs snippet against a sample app

Open
#2 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
typescript

Research direction

Start with src/tests/consumer-install.e2e.test.ts, docs/testing.md, and the sample-apps/issue-panel-uikit fixture. Trace the documented setup and determine how the separate consumer e2e script should pack and install the package before running the sample app tests. Done means the acceptance criteria pass, including the basic install flow, documented configuration, resource rendering, and the designated CI or pre-publish job.

Written by the indexing model from the issue text.

Description

enhancement

Problem

We have ~1194 tests covering the in-process API, MCP tools, manifest parsing, renderer, etc — but zero tests of the consumer install path. Every test runs from inside this repo, using local file paths (src/shims/forge-api.ts). No test simulates "user runs npm install forge-sim, follows the docs verbatim, and runs vitest."

This is a dogfooding gap. Three recent agent-driven audits surfaced consumer-perspective bugs that our internal suite is structurally incapable of catching:

Bug What our internal tests miss
require.resolve('forge-sim') in docs/testing.md throws ERR_PACKAGE_PATH_NOT_EXPORTED on every modern Node Our own vitest.config.ts aliases to src/shims/*.ts directly — never goes through the public exports map
.jsx files in resources fail with Unknown file extension ".jsx" at render time Our fixtures use .tsx everywhere
loadedResources: [] empty in forge_deploy response even when resources registered Internal tests don't assert against the public deploy response shape
Self-emitted .forge-sim/tsconfig.check.json produces TS5110 on every deploy Noise we tolerate internally; consumers see it on first try
@forge/react shim doesn't export List/ListItem despite being in UIKit 2 We don't test against the real UIKit 2 component surface

The vitest-config bug is the worst of these — it makes the test phase of the recommended workflow literally non-functional for new users. Fixed in #[TBD] with a drift-detection test, but that's a point fix. The next consumer-path bug will still slip through.

Proposed fix

A consumer-install e2e test, gated to a separate npm script (slow, not part of default vitest run):

// src/__tests__/consumer-install.e2e.test.ts
beforeAll(async () => {
  // 1. `npm pack` the current package into a temp dir
  // 2. mkdir a sample-app fixture (copy from sample-apps/issue-panel-uikit or similar)
  // 3. `npm install ./forge-sim-x.y.z.tgz` inside the sample app
  // 4. Write vitest.config.ts using the EXACT docs snippet
  //    (extracted from docs/testing.md, or maintained as a fixture that
  //    the existing drift detection test verifies matches the docs)
  // 5. Run `npx vitest run` inside the sample app
  // 6. Assert exit code 0
});

it('consumer can install, configure, and run a basic test following the docs');
it('consumer can render a UIKit panel with .tsx');
it('consumer can render a UIKit panel with .jsx');  // currently broken, but should work
it('forge_deploy response shape matches what the docs claim');
Why this is the right structural fix

Today's docs snippet drift-detection test catches this specific bug. The consumer-install e2e catches the entire class — any time the published package interface diverges from what the docs claim, the test screams. It also catches packaging regressions (missing files in dist/, broken exports map entries, etc.) that no unit test can reach.

Cost / gating

  • Pack + install dominates runtime; expect 30-60s per run
  • Gate to its own npm script (e.g. npm run test:e2e:consumer) and a separate CI job — not the default vitest run
  • Pre-publish hook is the right home

Acceptance criteria

  • A test that does npm pack + npm install ./tgz + writes a vitest config from the docs snippet + runs a basic test, gated behind its own npm script
  • Sample app fixture with at least one .tsx resource, ideally one .jsx resource (once the JSX bug is fixed)
  • Catches the original vitest-config bug if it regressed (sanity check: temporarily revert the docs fix, this test fails)
  • Catches the .jsx resource bug (related issue: TBD)
  • Runs in CI on a pre-publish or release workflow, not on every PR

Related

  • The drift detection point fix is in [the PR fixing the docs snippet]
  • This issue is the structural follow-up that catches the next bug like it
  • forge-sim is not yet published to npm — this test will need adapting once it is (or vice versa: this test gates the publish)
Dominant language
TypeScript
Stars
5
Forks
2
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ryanackley/forge-sim

All issues in ryanackley/forge-sim

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.