Add consumer-install e2e test: pack + install + run docs snippet against a sample app
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- typescript
- Domain
- build-system, ci-cd, testing
Research direction
Start with src/tests/consumer-install.e2e.test.ts, docs/testing.md, and the sample-apps/issue-panel-uikit fixture. Trace the documented setup and determine how the separate consumer e2e script should pack and install the package before running the sample app tests. Done means the acceptance criteria pass, including the basic install flow, documented configuration, resource rendering, and the designated CI or pre-publish job.
Written by the indexing model from the issue text.
Description
Problem
We have ~1194 tests covering the in-process API, MCP tools, manifest parsing, renderer, etc — but zero tests of the consumer install path. Every test runs from inside this repo, using local file paths (src/shims/forge-api.ts). No test simulates "user runs npm install forge-sim, follows the docs verbatim, and runs vitest."
This is a dogfooding gap. Three recent agent-driven audits surfaced consumer-perspective bugs that our internal suite is structurally incapable of catching:
| Bug | What our internal tests miss |
|---|---|
require.resolve('forge-sim') in docs/testing.md throws ERR_PACKAGE_PATH_NOT_EXPORTED on every modern Node |
Our own vitest.config.ts aliases to src/shims/*.ts directly — never goes through the public exports map |
.jsx files in resources fail with Unknown file extension ".jsx" at render time |
Our fixtures use .tsx everywhere |
loadedResources: [] empty in forge_deploy response even when resources registered |
Internal tests don't assert against the public deploy response shape |
Self-emitted .forge-sim/tsconfig.check.json produces TS5110 on every deploy |
Noise we tolerate internally; consumers see it on first try |
@forge/react shim doesn't export List/ListItem despite being in UIKit 2 |
We don't test against the real UIKit 2 component surface |
The vitest-config bug is the worst of these — it makes the test phase of the recommended workflow literally non-functional for new users. Fixed in #[TBD] with a drift-detection test, but that's a point fix. The next consumer-path bug will still slip through.
Proposed fix
A consumer-install e2e test, gated to a separate npm script (slow, not part of default vitest run):
// src/__tests__/consumer-install.e2e.test.ts
beforeAll(async () => {
// 1. `npm pack` the current package into a temp dir
// 2. mkdir a sample-app fixture (copy from sample-apps/issue-panel-uikit or similar)
// 3. `npm install ./forge-sim-x.y.z.tgz` inside the sample app
// 4. Write vitest.config.ts using the EXACT docs snippet
// (extracted from docs/testing.md, or maintained as a fixture that
// the existing drift detection test verifies matches the docs)
// 5. Run `npx vitest run` inside the sample app
// 6. Assert exit code 0
});
it('consumer can install, configure, and run a basic test following the docs');
it('consumer can render a UIKit panel with .tsx');
it('consumer can render a UIKit panel with .jsx'); // currently broken, but should work
it('forge_deploy response shape matches what the docs claim');
Why this is the right structural fix
Today's docs snippet drift-detection test catches this specific bug. The consumer-install e2e catches the entire class — any time the published package interface diverges from what the docs claim, the test screams. It also catches packaging regressions (missing files in dist/, broken exports map entries, etc.) that no unit test can reach.
Cost / gating
- Pack + install dominates runtime; expect 30-60s per run
- Gate to its own npm script (e.g.
npm run test:e2e:consumer) and a separate CI job — not the defaultvitest run - Pre-publish hook is the right home
Acceptance criteria
- A test that does
npm pack+npm install ./tgz+ writes a vitest config from the docs snippet + runs a basic test, gated behind its own npm script - Sample app fixture with at least one
.tsxresource, ideally one.jsxresource (once the JSX bug is fixed) - Catches the original vitest-config bug if it regressed (sanity check: temporarily revert the docs fix, this test fails)
- Catches the
.jsxresource bug (related issue: TBD) - Runs in CI on a
pre-publishorreleaseworkflow, not on every PR
Related
- The drift detection point fix is in [the PR fixing the docs snippet]
- This issue is the structural follow-up that catches the next bug like it
forge-simis not yet published to npm — this test will need adapting once it is (or vice versa: this test gates the publish)
- Dominant language
- TypeScript
- Stars
- 5
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ryanackley/forge-sim
-
upstream-changelog
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ryanackley/forge-sim#36 ·
-
upstream-changelog
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ryanackley/forge-sim#34 ·
-
upstream-changelog
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ryanackley/forge-sim#21 ·
-
upstream-changelog
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ryanackley/forge-sim#18 ·
-
upstream-changelog
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ryanackley/forge-sim#12 ·
All issues in ryanackley/forge-sim
Similar issues
-
clawsweeper:fix-shape-clear clawsweeper:queueable-fix clawsweeper:source-repro impact:ux-friction issue-rating: 🦞 diamond lobster no-stale P3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 76/100
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
github/gh-aw-firewall#8816 ·
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
vercel/react-tweet#225 ·