npm / npm/cli

[BUG] Big performance issues and possible memory leak

Open
#3,208 27 comments 28 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug perf Priority 1 Release 7.x Release 8.x
Dominant language
JavaScript
Stars
10.1k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
19

Description

Current Behavior:

Running npm 7 in docker image has some serious performance issues. Running same npm 7 locally doesn't have this on the other hand. I was thinking about posting this in the docker repository, but I am still unsure of the culprit and how it is related to the problem. I have found that reify:createSparse is the task that slows down dramatically, which as I saw is creating directories. So I suspect this issue might be related to docker volumes. On the other hand using npm@6 there is no such issue in docker.

Now when running same stuff on node:12-buster we also get the following out of heap error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory. This doesn't happen on node 14 and 16. (* actually just happend so I might be wrong here)

Originally I have found this issue using an ubuntu 16 docker image, but these other images also fail in the same manner.

Expected Behavior:

No performance issues in docker (vs local). No out of heap errors.

Steps To Reproduce:

Test package.json

{
  "name": "test",
  "version": "1.0.0",
  "main": "index.tsx",
  "scripts": {
  },
  "license": "ISC",
  "devDependencies": {
    "@cypress/code-coverage": "^3.9.5",
    "@cypress/webpack-preprocessor": "^4.1.5",
    "@storybook/addon-actions": "^6.1.11",
    "@storybook/addon-essentials": "^6.1.11",
    "@storybook/addon-knobs": "^6.1.11",
    "@storybook/addon-links": "^6.1.11",
    "@storybook/react": "^6.1.11",
    "apollo": "^2.32.5",
    "axios-mock-adapter": "^1.19.0",
    "babel-preset-react-app": "^10.0.0",
    "chai-in-viewport": "^1.0.3",
    "copy-webpack-plugin": "6.2.1",
    "cypress": "^7.2.0",
    "cypress-terminal-report": "^3.1.0",
    "fork-ts-checker-webpack-plugin": "4.0.3",
    "http-serve": "^1.0.1",
    "spa-http-server": "^0.9.0",
    "storybook-addon-designs": "^5.4.3",
    "storybook-addon-intl": "^2.4.1",
    "storybook-react-router": "^1.0.8"
  },
  "dependencies": {
    "@apollo/client": "^3.3.15",
    "@babel/core": "7.10.4",
    "@babel/plugin-syntax-dynamic-import": "7.8.3",
    "@babel/preset-env": "7.10.4",
    "@babel/preset-react": "7.10.4",
    "@babel/register": "^7.10.4",
    "@brokerloop/ttlcache": "^3.2.1",
    "@hot-loader/react-dom": "^16.12.0",
    "@material-ui/core": "5.0.0-alpha.10",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "4.0.0-alpha.45",
    "@reduxjs/toolkit": "^1.5.0",
    "@types/axios": "^0.14.0",
    "@types/chai": "^4.2.8",
    "@types/chai-as-promised": "^7.1.2",
    "@types/mocha": "^7.0.1",
    "@types/node": "12.0.2",
    "@types/react": "^16.8.23",
    "@types/react-dom": "^16.0.11",
    "@types/react-redux": "^7.1.1",
    "@types/react-router": "^5.1.4",
    "@types/react-router-dom": "^5.1.3",
    "@types/react-slick": "^0.23.4",
    "@types/react-stickynode": "^2.1.0",
    "@types/redux-mock-store": "^1.0.2",
    "@types/sinon": "^7.5.1",
    "@types/storybook-react-router": "^1.0.1",
    "@types/throttle-debounce": "^2.1.0",
    "@types/url-join": "^4.0.0",
    "@types/uuid": "^8.3.0",
    "@types/yup": "^0.26.33",
    "@xstate/react": "^0.8.1",
    "autoprefixer": "^9.7.5",
    "axios": "0.18.1",
    "babel-loader": "8.0.6",
    "chai": "^4.2.0",
    "chai-as-promised": "^7.1.1",
    "clean-webpack-plugin": "^3.0.0",
    "clone-deep": "^4.0.1",
    "clsx": "^1.1.1",
    "css-hot-loader": "^1.4.4",
    "css-loader": "3.4.2",
    "currency-symbol-map": "^4.0.4",
    "deep-equal": "^2.0.2",
    "deepmerge": "^4.2.2",
    "dotenv": "8.2.0",
    "dotenv-webpack": "1.7.0",
    "envsub": "^4.0.7",
    "fast-deep-equal": "^3.1.3",
    "file-loader": "^5.0.2",
    "formik": "^2.1.4",
    "graphql": "^15.5.0",
    "html-webpack-harddisk-plugin": "^1.0.2",
    "html-webpack-plugin": "^4.3.0",
    "istanbul-instrumenter-loader": "^3.0.1",
    "istanbul-lib-coverage": "^3.0.0",
    "js-cookie": "^2.2.1",
    "jsdom": "16.1.0",
    "jsdom-global": "3.0.2",
    "jsonschema": "^1.2.5",
    "jss-increase-specificity": "^0.3.5",
    "material-ui-image": "^3.2.3",
    "md5-file": "^4.0.0",
    "mini-css-extract-plugin": "^0.9.0",
    "mocha": "^7.0.1",
    "nyc": "^15.0.0",
    "postcss-loader": "3.0.0",
    "prettier": "^1.18.2",
    "query-string": "^6.11.0",
    "react": "16.12.0",
    "react-dom": "16.12.0",
    "react-hot-loader": "^4.12.19",
    "react-intl": "^3.12.0",
    "react-router": "^5.1.2",
    "react-router-dom": "^5.1.2",
    "react-scrollbars-custom": "^4.0.25",
    "react-slick": "^0.27.11",
    "react-stickynode": "^2.1.1",
    "react-universal-component": "^4.0.1",
    "redux-mock-store": "^1.5.4",
    "remove": "^0.1.5",
    "sass-loader": "8.0.2",
    "sinon": "^8.1.1",
    "slick-carousel": "^1.8.1",
    "thread-loader": "^3.0.3",
    "throttle-debounce": "^3.0.1",
    "tinycolor2": "^1.4.1",
    "tinyurl": "^1.1.7",
    "ts-loader": "6.2.1",
    "ts-node": "^8.6.2",
    "tslint": "^5.12.1",
    "tslint-config-prettier": "^1.17.0",
    "tslint-config-standard": "^9.0.0",
    "tslint-no-focused-test": "^0.5.0",
    "tslint-react": "^4.0.0",
    "typescript": "3.7.5",
    "url-join": "^4.0.1",
    "uuid": "^8.3.1",
    "webpack": "4.46.0",
    "webpack-cli": "3.3.10",
    "webpack-dev-server": "3.10.3",
    "webpack-license-plugin": "^4.1.1",
    "webpack-livereload-plugin": "^2.2.0",
    "webpack-merge": "4.2.2",
    "xstate": "^4.7.8",
    "yup": "^0.28.3"
  },
  "engines": {
    "node": ">=12"
  }
}

Example 1.

  1. Do a local npm i to create the package lock in the directory where you created the above package.json.
  2. Run rm node_modules/ -rf
  3. Run docker run --rm -it -v $PWD:/app -w /app node:16-buster npm i --legacy-peer-deps --verbose in the directory where you created the above package.json
  4. Notice how long the reify:createSparse
  5. Same behaviour is with node:14-buster

Example 2:

  1. Do a local npm i to create the package lock in the directory where you created the above package.json.
  2. Run rm node_modules/ -rf
  3. Run docker run --rm -it -v $PWD:/app -w /app node:12-buster bash -c "npm i -g npm@7 && npm i --legacy-peer-deps --verbose"
  4. Notice the out of heap error.
Environment:
  • Host OS: Ubuntu 20.04 Kernel: 5.11.10-051110-generic
  • Docker: 20.10.6
  • Node: 16.x, 14.x, 12.x
  • npm: 7.x
  • System RAM: 32GB, CPU i7-10850H

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Docker-mounted install with the supplied package.json and commands, comparing npm 7 with npm 6 and the listed Node versions. Trace the reify:createSparse task, which the report identifies as the slowdown point, while checking the reported heap failure. Done means the install no longer shows the Docker performance regression or out-of-heap error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, javascript, node.js
Domain
cli, devtools, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.