npm / npm/cli

[BUG] npm exec --workspaces very slow

Open
#5,509 5 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug config:workspaces perf Release 8.x ws:libnpmexec
Dominant language
JavaScript
Stars
10.1k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
19

Description

Is there an existing issue for this?
  • I have searched the existing issues
This issue exists in the latest npm version
  • I am using the latest npm
Current Behavior

We have a monorepo using npm workspaces, containing ~70 workspaces/packages.

We noticed that running npm exec --workspaces (for example, npm exec --workspaces --call 'pwd') at the top-level of this monorepo was incredibly slow - the npm process would be pegged at 100% CPU and it would take multiple seconds to print the directory path for each workspace.

The overall run time of npm exec --workspaces --call 'pwd' can take several minutes.

I was able to bisect NPM release versions to identify that this command was fast in version 8.7.0, but slow in version 8.8.0, so it appears that https://github.com/npm/cli/pull/4643 introduced the slowness (cc @ruyadorno). Indeed if I locally revert the change from that PR (modified revert due to subsequent changes from #5250), the performance issue disappears:

--- /Users/gavin/.nvm/versions/node/v16.17.0/lib/node_modules/npm/lib/commands/exec.js	2022-09-14 15:12:56.000000000 -0400
+++ /Users/gavin/.nvm/versions/node/v16.17.0/lib/node_modules/npm/lib/commands/exec-2.js	2022-09-14 15:12:37.000000000 -0400
@@ -25,7 +25,7 @@
 
   async exec (_args, { locationMsg, runPath } = {}) {
     // This is where libnpmexec will look for locally installed packages
-    const localPrefix = this.npm.localPrefix
+    const localPrefix = runPath ? runPath : this.npm.localPrefix
Expected Behavior

npm exec --workspaces --call 'pwd' should very quickly (<2s) print out all workspace directories.

Steps To Reproduce
  1. In a large monorepo with ~70 workspaces
  2. With a default workspaces config
  3. Run npm exec --workspaces --call 'pwd'
  4. See slowness
Environment
  • npm: 8.19.2
  • Node.js: 16.17.0
  • OS Name: macOS 12.5.1
  • System Model Name: Macbook Pro (2021 M1 Max)
  • npm config:
; node bin location = /Users/gavin/.nvm/versions/node/v16.17.0/bin/node
; node version = v16.17.0
; npm local prefix = /Users/gavin/dev/league-web
; npm version = 8.19.2
; cwd = /Users/gavin/dev/league-web
; HOME = /Users/gavin

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lib/commands/exec.js and the change introduced by pull request #4643, then reproduce npm exec --workspaces --call 'pwd' in a monorepo with about 70 workspaces. Compare the behavior with npm 8.7.0 and the reported revert; done means all workspace directories print in under two seconds without pegging the CPU.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.