Proposal: Split ESM build work into a dedicated unpkg-build service

Summary

Status: Proposal — This approach is being proposed for evaluation and is not yet a committed implementation.

This issue proposes creating a new Bun/Fly service, packages/unpkg-build, and moving ESM build and inline transform work out of unpkg-files into it.

This is a workload-isolation change, not a caching or build-pipeline redesign. Preserve the existing behavior: unpkg-build should fetch npm tarballs and perform builds on demand exactly as unpkg-files does today. Do not introduce a local cache, object store, artifact persistence, build queue, or cross-request deduplication in this issue.

Why

unpkg-files currently serves two different workload classes from the same Fly process:

• latency-sensitive immutable package file reads and listings;
• CPU-, memory-, and disk-heavy ESM builds and inline TypeScript/TSX transforms.

Separating these workloads may help alleviate some pressure on the existing origin servers by reducing contention between build traffic and core package-file traffic. It would also give the build workload an independent deployment and scaling boundary without changing public URL behavior. The operational benefit should be validated with production metrics before and after rollout.

Decisions already made

• Introduce a separate unpkg-build Fly service.
• Route by operation, not dynamically by load:
  • /file/* and /list/* belong to unpkg-files.
  • /build/* and POST /transform belong to unpkg-build.
  • Both services expose /_health.
• unpkg-build reads npm tarballs directly through shared Bun-only tarball code. It must not call the public unpkg-files HTTP service to obtain build inputs.
• Preserve the current esbuild implementation, options, headers, status codes, and diagnostics unless a change is strictly required by the split.
• Use port 4000 for local unpkg-files and port 4001 for local unpkg-build.
• Add BUILD_ORIGIN to unpkg-esm; retain FILES_ORIGIN for raw package-file access.

Explicit non-goals

Do not add any of the following as part of this work:

• local package or artifact caching;
• Tigris, R2, Fly Volumes, or another object store;
• durable build artifact persistence;
• a build queue or background jobs;
• global or process-local singleflight/deduplication;
• rate limiting or a new authentication scheme;
• bundler replacement or ESM compatibility changes;
• a general service-contract rewrite;
• changes to public unpkg.com, app.unpkg.com, or esm.unpkg.com URL semantics.

Those can be evaluated separately using production measurements after the service boundary exists.

Target architecture

unpkg-www  ─┐
             ├─ /file, /list ───────> unpkg-files (Bun/Fly) ──> npm
unpkg-app  ─┘

unpkg-esm ─── /file ────────────────> unpkg-files (Bun/Fly) ──> npm
          └── /build, /transform ───> unpkg-build (Bun/Fly) ──> npm

Cloudflare caching and all public request routing remain where they are today. This issue changes only the origin selected by unpkg-esm for build/transform requests.

Package and code organization

Add packages/unpkg-build

Create a private workspace package consistent with unpkg-files, including:

• package.json;
• tsconfig.json and tsconfig.build.json;
• src/server.ts;
• an exported request handler for tests;
• src/lib/request-handler.ts;
• the existing ESM build service and its tests;
• request logging equivalent to the current Bun origin logging;
• Dockerfile;
• production and staging Fly configuration;
• README.md.

Suggested scripts:

{
  "build": "tsc --project ./tsconfig.build.json",
  "dev": "MODE=development bun --port 4001 ./src/server.ts",
  "test": "bun --preload=./test/setup.ts test",
  "deploy": "fly deploy ../.. -c ./packages/unpkg-build/fly.json",
  "deploy:staging": "fly deploy ../.. -c ./packages/unpkg-build/fly.staging.json"
}

The initial Fly machine/runtime settings should mirror unpkg-files unless the deployment owner chooses different capacity during rollout. The important requirement for this issue is a separate Fly app/process and scaling boundary, not capacity tuning.

Share Bun-only npm tarball access

Both services need the existing getFile, listFiles, withPackageFileDirectory, PackageNotFoundError, and TarballFetchTimeoutError behavior from packages/unpkg-files/src/lib/npm-files.ts.

Do not duplicate that implementation and do not move Node/Bun filesystem or stream code into unpkg-worker, which is consumed by Cloudflare Workers.

Extract the existing tarball reader into a small Bun-only workspace library (suggested name: packages/unpkg-npm) and have both unpkg-files and unpkg-build depend on it. Move the directly supporting content-type and integrity code/tests with it as needed. Keep this extraction mechanical: preserve current network, extraction, timeout, error, content-type, and integrity behavior.

Expected dependency direction:

unpkg-files ─┐
              ├──> unpkg-npm ──> unpkg-worker domain types
unpkg-build ─┘

If implementation constraints reveal a simpler non-duplicating Bun-only library boundary, document it in the PR, but do not make one deployable call the other over HTTP.

Reduce unpkg-files to file responsibilities

After the move, unpkg-files should handle only:

• GET/HEAD /file/:package@:version/:path;
• GET/HEAD /list/:package@:version/:prefix?;
• GET/HEAD /_health;
• existing method handling, logging, and file-origin error mapping.

Remove /build and /transform handling and remove build-only dependencies such as esbuild and es-module-lexer from unpkg-files when they are no longer needed there.

Build service endpoints

unpkg-build should handle only:

• GET/HEAD /build/:package@:version/:path?;
• POST /transform;
• OPTIONS as required by the current behavior;
• GET/HEAD /_health.

Preserve:

• query option parsing through normalizeBuildOptions;
• semver validation;
• current build result headers, including X-UNPKG-* headers;
• existing 404, 415, 422, 500, and 504 behavior;
• HEAD response behavior;
• current npm registry origin and tarball timeout behavior;
• current synchronous request lifecycle—no queue or background continuation.

Requests for /file or /list on unpkg-build, and /build or /transform on unpkg-files, should return 404/method responses rather than proxying between services.

unpkg-esm routing changes

Extend packages/unpkg-esm/src/env.ts with:

BUILD_ORIGIN: string;

Update all environments in packages/unpkg-esm/wrangler.json:

• development: http://localhost:4001;
• staging: the staging unpkg-build Fly origin;
• production: the production unpkg-build Fly origin.

Keep FILES_ORIGIN unchanged for raw file requests.

Update call sites as follows:

• normal ESM artifact build requests use BUILD_ORIGIN;
• ?meta build/integrity requests use BUILD_ORIGIN;
• POST /transform proxying uses BUILD_ORIGIN;
• raw files, declaration files, and CSS source reads continue using FILES_ORIGIN.

Do not add fallback from BUILD_ORIGIN to FILES_ORIGIN; the services have different contracts.

Update tests so their environment fixtures include both origins and their fetch dispatchers route each origin to the appropriate in-process handler.

Repository tooling and local development

Update the root and project documentation:

• root README.md package list, architecture description, commands, and ports;
• AGENTS.md repository structure and common development commands;
• ESM architecture documentation where it still describes builds as part of unpkg-files;
• add pnpm --filter unpkg-build dev to local startup instructions.

Add root convenience scripts consistent with the existing naming, including at minimum:

• deploy:build;
• a staging build deploy command, or inclusion in an appropriately named origin staging command.

Do not silently add unpkg-build to deploy:workers; it is a Fly service, not a Cloudflare Worker.

Ensure both Docker builds include and build the new shared Bun-only package. Keep production images limited to the packages and runtime dependencies each service needs.

CI and compatibility tooling

Update .github/workflows/ci.yml so the ESM smoke job:

1. starts unpkg-files on port 4000;
2. starts unpkg-build on port 4001;
3. starts unpkg-esm with its development environment;
4. waits for both Bun health endpoints and the Worker endpoints;
5. prints the new service log on readiness or test failure;
6. cleans up the new process reliably.

Update local ESM compatibility/service-management tooling in scripts/esm-compat-suite.ts and related smoke/readiness scripts where they assume that build traffic is served by unpkg-files. Local restart/recovery logic must understand the additional service without changing corpus semantics.

Test migration and additions

• Move esm-build-service.test.ts to unpkg-build with the implementation.
• Move build and transform request-handler tests from unpkg-files to unpkg-build.
• Keep /file and /list request tests in unpkg-files.
• Move tarball-reader unit tests to the shared Bun-only package if that code is extracted there.
• Add negative routing tests proving the services do not proxy or accidentally retain one another's endpoints.
• Update unpkg-esm integration tests to use separate file and build origins.
• Run the existing seed compatibility and browser smoke suites against the four-service local topology.

Deployment and rollout order

The repository changes should support this safe rollout:

1. Provision and deploy production/staging unpkg-build services while unpkg-files still supports the old build endpoints.
2. Verify /_health and direct representative /build and /transform requests.
3. Deploy unpkg-esm with BUILD_ORIGIN pointing to unpkg-build.
4. Run the seed compatibility and browser smoke suites against staging/production as appropriate.
5. Confirm build traffic has moved and file traffic remains healthy.
6. Deploy the version of unpkg-files that removes /build and /transform.

Keep the temporary backward-compatible deployment ordering explicit in the PR/release notes. Do not remove the old endpoints before unpkg-esm is using the new service.

Acceptance criteria

• packages/unpkg-build exists as an independently buildable, testable, runnable, and deployable Bun/Fly service.
• unpkg-files no longer contains ESM build implementation code or serves /build and /transform.
• unpkg-build serves /build, POST /transform, and /_health, but not /file or /list.
• The npm tarball reader is shared as Bun-only library code without duplication or service-to-service HTTP calls.
• unpkg-esm uses BUILD_ORIGIN for every build/transform path and FILES_ORIGIN for every raw file path.
• Development uses ports 4000 and 4001 for file and build services respectively.
• Production and staging Fly configurations and deployment scripts exist for unpkg-build.
• CI starts and health-checks the new service and runs existing ESM compatibility/browser smoke coverage through it.
• Existing public response status codes, headers, redirects, diagnostics, and generated module output remain compatible.
• The rollout can be performed in the documented order without a window where ESM build requests have no origin.
• No cache, object store, artifact persistence, queue, or deduplication mechanism is introduced.

Validation commands

At minimum, run:

pnpm install
pnpm run build
pnpm test
pnpm test:esm-compat
pnpm test:esm-browser -- --origin http://localhost:3002 --run-origin http://localhost:3000

For local service validation, start:

pnpm --filter unpkg-files dev
pnpm --filter unpkg-build dev
pnpm --filter unpkg-www dev
pnpm --filter unpkg-esm dev

Then verify:

curl --fail http://localhost:4000/_health
curl --fail http://localhost:4001/_health
curl --fail 'http://localhost:4000/file/react@18.2.0/package.json'
curl --fail 'http://localhost:4001/build/react@18.2.0?target=es2022'

Start with packages/unpkg-files, packages/unpkg-esm, and the proposed packages/unpkg-build and packages/unpkg-npm boundaries; read the existing request handlers, tarball reader, and ESM routing tests first. Run the listed build and test commands, then validate the four-service topology and CI changes. Done means the services, shared Bun-only library, routing, deployment files, tests, and rollout behavior meet the acceptance criteria without changing public responses.