HarperFast / HarperFast/studio
Download Application: consume the streamed archive once harper#2152 ships
- Dominant language
- TypeScript
- Stars
- 5
- Forks
- 4
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 40
Description
Follow-up work unlocked once HarperFast/harper#2152 lands and reaches instances. Blocked until then.
## Background
#1591 was a browser crash downloading a large application. #1617 shipped the Studio-side guardrail — the modal states the package size up front and cautions when it's large or unmeasurable — but deliberately did **not** change the transport, because verifying that needs a Harper build carrying #2152.
harper#2152 (closes HarperFast/harper#2150) adds two things to `package_component`:
- `stream: true` — returns the tar.gz as raw bytes with `content-type: application/gzip` and a `content-disposition` filename, instead of base64 inside JSON. Constant memory, no size ceiling.
- `estimate: true` — returns `{ project, total_size, dangling_symlinks }` from a single directory walk, packaging nothing.
The base64 shape stays the default, so nothing breaks on its own.
## 1. Consume the streamed archive
The reason this is worth doing: today's path makes ~5 full-size copies of the archive in the renderer (response text → `JSON.parse` → `atob` → byte array → Blob), and both strings hit V8's 512 MiB cap. Streaming removes all of it.
- [ ] Move this one call off axios. It's XHR, so it buffers the whole body and defeats the point. `resolveInstanceConnection` already resolves URL + auth + `mode` for exactly this purpose (`deployComponentStream` uses it).
- [ ] Send `stream: true` and **branch on the response `content-type`** — `application/gzip` → stream path, JSON → the existing base64 path. One code path covers old and new instances with no version check needed.
- [ ] Write to disk: `showSaveFilePicker()` + `response.body.pipeTo(writable)` where available (Chromium), falling back to `await response.blob()` elsewhere. The fallback is still a large win — Chrome backs big Blobs with disk, so it sidesteps the string cap, `atob`, and the per-byte decode loop even without File System Access.
- [ ] Handle `mode: 'proxy'` explicitly. Fabric Connect is a message-passing proxy with a 2 MB body cap ("Fabric Connect body size must be less than 2MB. Utilize Direct Connect for larger payloads.") and cannot stream at all, so proxy-mode instances need a clear message rather than a hang or a truncated file.
**Unknown to settle first:** whether the download currently goes direct or through the Fabric Connect proxy. The 5 MB app used to verify #1617 downloaded fine, but its base64 payload was probably under the 2 MB cap, so that run doesn't distinguish the two. This decides how prominent the proxy handling needs to be.
## 2. Use `estimate` for the include-node_modules case — ⚠️ needs a version gate
`get_components` omits `node_modules`, so with "Include Node Modules" ticked Studio can't size the package and currently just cautions unconditionally. `estimate` would give it a real number.
**It cannot be called blindly.** Harper's operation validator runs with `allowUnknown: true` and the pre-#2152 handler ignores fields it doesn't recognise — so on an older instance `package_component { project, estimate: true }` doesn't return an estimate, it **packages the entire application and base64-encodes it**. Asking "how big is this?" would trigger the exact expensive operation the estimate exists to avoid, on precisely the applications where that hurts most.
Note the asymmetry, because it affects sequencing: `stream: true` degrades harmlessly on an old instance (unknown key ignored → JSON response → the content-type branch handles it), while `estimate: true` degrades badly. **Item 1 can ship on content-type sniffing alone; item 2 needs a real capability or version check.**
## 3. Recalibrate the caution once streaming is live
- [ ] The warning currently says "Studio has to hold the whole archive in browser memory before it can be saved." That stops being true on the `showSaveFilePicker` path and is much weaker on the Blob path. Both the copy and `LARGE_PACKAGE_BYTES` should become conditional on the transport actually chosen — otherwise we're warning about a problem we fixed.
- [ ] The `unmeasured` caution stays useful regardless: it covers instances that report no file sizes, which are also the instances least likely to have the streamed download.
## 4. Minor: the download filename
`DownloadApplicationModal` names the file `${project}.gz`, but the archive is a tar inside a gzip. The stream path fixes itself — Harper sets `filename="${project}.tar.gz"` via `content-disposition` — so only the legacy base64 path would still be wrong.
## Not planned
Multi-file download (one request per file, reassembled client-side) was considered and rejected in HarperFast/harper#2150: the browser can't write a directory tree without `showDirectoryPicker()`, a real application is thousands of extra round trips through `get_component_file`'s own base64-in-JSON, it discards symlink/mode/empty-dir fidelity, and reassembling into one artifact means zipping in the browser — pushing every byte back through JS memory, which is the thing being fixed.
Contributor guide
Research direction
Wait for HarperFast/harper#2152 to reach instances, then start at DownloadApplicationModal and trace the current download through resolveInstanceConnection and deployComponentStream. Determine whether downloads use direct or proxy mode, verify old JSON and new application/gzip responses, and confirm that streaming, fallback saving, proxy handling, estimates, warning text, and legacy filenames behave as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend, web-dev
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100