Automattic / Automattic/studio

Create-from eagerly base64-encodes captures and exceeds Node string limits

Open
#4,735 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
517
Forks
95
Avg merge
1d 22h
Merged PRs (30d)
162

Description

## Problem

The `studio create --from` implementation in PR #4725 eagerly reads every file from a directory source, base64-encodes it, and then serializes the complete payload more than once while constructing the SSI request.

Real acceptance with a 517 MB Data Liberation capture fails with `Invalid string length` before WordPress is created. Base64 expansion alone pushes the request above Node’s practical string ceiling. Buffering WP-CLI output was initially suspected but is not involved in this failure.

## Root cause

`resolveStaticSiteImporterSource()` calls `collectSourceFiles()`, which calls `fs.readFileSync(...).toString("base64")` for every source file. `buildStaticSiteImporterRequest()` then runs `JSON.stringify(payload)` for provenance and `buildCreateFromSourceBlueprint()` serializes the complete request.

## Expected behavior

- Source bytes remain on disk and are streamed/staged rather than represented in an in-memory JSON control message.
- Studio creates a bounded request that references an archive staged inside the target site/import workspace through SSI’s server-owned resolver contract.
- Failed imports preserve both the identical request and its staged source for deterministic resume.
- Successful imports remove staged source bytes.
- Tests prove the request size is independent of source payload size and real create-from acceptance succeeds with a large capture.

## Upstream dependency

SSI needs a generic request-bundle source reference resolver; its tracker will be linked here after creation.

## AI assistance

OpenAI GPT-5.6 Sol was used through OpenCode to run acceptance, inspect the 517 MB fixture, and trace the exact eager base64 and serialization path. Chris directed and reviewed the investigation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing PR #4725 through resolveStaticSiteImporterSource(), collectSourceFiles(), buildStaticImporterRequest(), and buildCreateFromSourceBlueprint(), confirming where base64 data and repeated JSON serialization inflate the request. Coordinate the SSI request-bundle source reference resolver, then verify bounded request size, deterministic resume with preserved staged bytes, cleanup after success, and successful large-capture acceptance.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend-api-design, cli, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.