yarnpkg / yarnpkg/berry

[Bug?]: workspaces run concurrency

Open
#3,901 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug reproducible
Dominant language
TypeScript
Stars
8.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Self-service
  • I'd be willing to implement a fix
Describe the bug

See the following command, where as I understand should run the build script in all the packages listed in dependencies or devDependencies in topological order, but it fails if some of the packages mentioned has not a build script while it list other dependencies that should be build beforehand too.

yarn workspaces foreach --recursive --topological-dev --from packageA run build
packageA
  dependencies:
    packageB
      dependencies:
        packageC
          scripts:
            build: "tsc ..."
  scripts:
     build: "tsc ..."

As packageB has not a build script it DOES NOT WAIT FOR the build of packageC going into a race condition.

To workaround, we can add to packageB a dumb script "build": "true". In my opinion, it is a bug, it does not match the expectation (see docs):

Yarn will only run the command after all workspaces that it depends on through the dependencies field have successfully finished executing.

To reproduce
const {promises: {readFile, writeFile}} = require("fs");
const path = require("path");

const file = path.resolve("output.log");

await writeFile(file, `
`);

await packageJson({
  name: "root",
  private: true,
  workspaces: ["packages/*"],
});

await packageJson(
  {
    name: "packageA",
    version: "1.0.0",
    dependencies: {
      "packageB": "*",
    },
    scripts: {
      build: "echo '** A **' >> " + file
    },
  },
  { cwd: "packages/packageA" }
);

await packageJson(
  {
    name: "packageB",
    version: "1.0.0",
    dependencies: {
      "packageC": "*",
    },
    // if the package has not the build script then it DOES NOT block
    // scripts: {
    //   build: "echo '** B **' >> " + file
    // },

    // a workaround is to use a dummy script, in this case it DOES block
    // scripts: {
    //   build: "true"
    // },
  },
  { cwd: "packages/packageB" }
);

await packageJson(
  {
    name: "packageC",
    version: "1.0.0",
    scripts: {
      build: "echo '** C **' >> " + file
    },
  },
  { cwd: "packages/packageC" }
);

await writeFile(".yarnrc.yml", `
  nodeLinker: node-modules
`);

await yarn("install");
await yarn("workspaces", "foreach", "--recursive",  "--topological-dev", "--from", "packageA", "run", "build");

const output = await readFile(file, "utf8");

// when packageB HAS NOT a build script, the topology IS NOT correct, building: A > C
expect(output).toContain(`
** C **
** A **
`);

// when packageB has a build script, the topology is correct, building: A < B < C
// expect(output).toContain(`
// ** C **
// ** B **
// ** A **
// `);
Environment
System:
    OS: macOS 12.1
    CPU: (10) arm64 Apple M1 Pro
  Binaries:
    Node: 16.13.1 - /private/var/folders/2t/vh754bvs45303cbdj6b063jc0000gp/T/xfs-e051e464/node
    Yarn: 3.1.1 - /private/var/folders/2t/vh754bvs45303cbdj6b063jc0000gp/T/xfs-e051e464/yarn
    npm: 8.1.2 - /opt/homebrew/Cellar/node@16/16.13.1/bin/npm
Additional context

If the build of packageC is very slow (imagine several minutes), we will be running code in packageA with stale version of packageC or without if it was cleaned early.

broken chain:

A -- import --> B -- import --> C
// packageA
import foo from 'packageB';
// packageB
export foo from 'packageC';
// packageC
export default 123;

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied JavaScript repro, the generated package.json files, .yarnrc.yml, and the yarn workspaces foreach --recursive --topological-dev entry point. Verify the ordering with packageA, packageB, and packageC when packageB lacks a build script; done means packageA waits for packageC as required by the documented dependency topology.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, typescript
Domain
build-system, cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.