posit-dev / posit-dev/positron

Builds: Compile built-in extensions only once (improve build speed & upstream alignment)

Open
#965 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: builds
Dominant language
TypeScript
Stars
4.3k
Forks
184
Avg merge
1d 11h
Merged PRs (30d)
206

Description

Currently, the Positron build scripts create a universal macOS binary by doing a full build of arm64 and a full build of x86, and then stitching them together.

This sort of works, but it requires some hacks. For example, some of the build artifacts aren't compatible with this approach because the process that produces them is not stable. In order for us to stitch together the two independent builds using create-universal-app, every file must match, or support being fused. This is the reason for this code, which, while not outright smelly, does at least have a very unappetizing aroma.

https://github.com/rstudio/positron/blob/c2d0ec0974d45eaa249515234bae4ee484138d66/.github/workflows/build-release-macos.yml#L201-L225

This isn't the way that VS Code works; its core TypeScript compilation step happens only once. The compilation results, which aren't architecture-specific, are cached and reused for the arm64 build, the x86 build, and the Universal build. Here is where the compilation artifact is downloaded:

https://github.com/rstudio/positron/blob/c2d0ec0974d45eaa249515234bae4ee484138d66/build/azure-pipelines/darwin/product-build-darwin.yml#L33-L40

We cannot do this in Positron, however, because the extensions that we have added contain binaries that are architecture dependent. For example the jupyter-adapter extension uses an architecture-specific build of ZeroMQ, and the positron-r extension contains an architecture-specific build of the ark kernel.

In VS Code 1.80, a change was made (which I have thus far been unable to pinpoint) which causes further divergence of the arm64 and x86 files, such that even the minified Javascript is different between the builds.

Error: Expected all non-binary files to have identical SHAs when creating a universal build but "Contents/Resources/app/extensions/emmet/dist/node/emmetNodeMain.js" did not
    at exports.makeUniversalApp (/Users/user229818/actions-runner/_work/positron/positron/build/node_modules/vscode-universal-bundler/dist/cjs/index.js:73:23)
    at runMicrotasks (<anonymous>)

In order to update to 1.80 and beyond, we need to sort out our build system so that it is more like VS Code's, while still accommodating our use case. Here are some ideas:

  • Put Positron's extensions into a separate build task and let that build task run per architecture. Keep VS Code's built-in extensions in their own build task that only runs once, just like it does in the upstream build system.
  • Let all the extensions share a build task as they do today. For the Universal build, however, don't try to fuse the output of two different compilations. Instead start with the output of the amd64 build task, but overlay the native code build for x64 right before producing a Universal app.
  • Adopt VS Code's build-once approach, producing a single compilation.tar.gz file. Hold ourselves to the same constraints VS Code does: do not include any native code in built-in extensions. Native binaries, like ark and zeromq, must be separately built and loaded into e.g. Resources.
  • Run away from home and live in the woods.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the macOS workflow section in .github/workflows/build-release-macos.yml with the upstream compilation artifact handling referenced in build/azure-pipelines/darwin/product-build-darwin.yml. First determine which built-in extensions and native binaries differ by architecture; done means built-in extensions compile once while architecture-specific binaries remain correctly included in the arm64, x86, and Universal builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, macos, typescript
Domain
build-system, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.