galaxyproject / galaxyproject/brc-analytics

Give BRC and GA2 builds separate output/staging dirs so they can build in parallel

Open
#1,622 1 comment 0 reactions 0 assignees View on GitHub
build
Dominant language
TypeScript
Stars
7
Forks
11
Avg merge
2d 12h
Merged PRs (30d)
16

Description

## Problem

The BRC Analytics and GA2 site builds (`npm run build:local` and `npm run build-local:ga2`) **cannot run in parallel** — they write to the same output and staging locations, so running them concurrently corrupts each other. Today they must run strictly sequentially (in CI, `run-checks.yml` runs `build:local` then `build-local:ga2` one after the other; locally, same). Since each build is a full `next build --webpack` static export, serial execution roughly doubles wall-clock build time for what are two independent sites.

Both build chains are:

```
./scripts/build.sh # stage env + favicons
&& ./scripts/set-version.sh # append version vars to the env file
&& ./scripts/sync-api-.sh # stage catalog JSON into public/api
&& next build --webpack # build + static export
```

## Why they collide (concrete shared targets)

Every build writes these **shared** paths, so a second build overwrites the first:

1. **`public/api/*.json`** — the biggest one. Both `scripts/sync-api-brc-analytics.sh` and `scripts/sync-api-ga2.sh` do `rm -f public/api/*.json` and then copy *their* site's catalog JSON into the same `public/api/`. A parallel run interleaves the delete/copy of the two sites and produces a mixed/partial API dir.
2. **`.env.production`** (repo root) — `scripts/build.sh` copies `site-config///.env` to a single root `.env.production`, and `scripts/set-version.sh` appends `NEXT_PUBLIC_*` to that same file. Two builds race on it.
3. **`public/favicons/`** — `scripts/build.sh` copies the per-site favicons into the shared `public/favicons/`.
4. **`.next/` and `out/`** — `next build` with `output: "export"` (see `next.config.mjs`) writes the build cache to the default `distDir` (`.next`) and the exported static site to `out/`. Both are shared and overwritten per build.

## Proposed change

Give each site build a fully isolated set of output/staging locations so the two can run concurrently. Two viable approaches:

- **A — per-site directories in one tree.** Parameterize the build on the site: per-site `distDir` (e.g. `.next-brc` / `.next-ga2`), per-site export/output dir (e.g. `out/brc-analytics` / `out/ga2`), per-site env file instead of a single root `.env.production`, and stage `api`/`favicons` into a per-site public/staging dir rather than the shared `public/`. Requires the app to resolve its public asset base per build.
- **B — build each site in its own git worktree/checkout.** Cheapest isolation: run each site's build in a separate working copy so all the shared paths above are naturally distinct. No app changes; mostly a CI/script change. Good if per-site `distDir`/public plumbing turns out to be invasive.

Then run the two builds concurrently:
- **CI:** split the GA2 build into its own parallel job (or `&`-parallelize within `run-checks.yml`) instead of running after the BRC build.
- **Local:** a script/target that runs both at once.

## Acceptance criteria

- [ ] `build:local` (BRC) and `build-local:ga2` (GA2) can run **concurrently** without clobbering each other's `public/api`, env, favicons, or build output.
- [ ] Each site's build output is byte-for-byte equivalent to today's sequential build.
- [ ] CI (`run-checks.yml`) builds the two sites in parallel; total checks wall-clock drops.
- [ ] Documented local command to build both at once.

## Related

Part of the GA2 / BRC site-separation effort (spike #1246). This is a natural follow-on now that the per-site `sites/*` split is landing (e.g. #1598 / #1599) — the code is separating; the build outputs should too.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with scripts/build.sh, scripts/set-version.sh, the two scripts/sync-api-*.sh files, next.config.mjs, and .github/workflows/run-checks.yml; run the existing BRC and GA2 build commands sequentially to establish their outputs. Compare concurrent results with sequential builds, then verify isolated API, environment, favicon, and build outputs, parallel CI execution, and the documented local command.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, next.js, shell, typescript
Domain
build-system, ci-cd, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.