Size of node_modules has considerable effect on build speed
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 9.8k
- Forks
- 326
- PR merge metrics
- No merged PRs in 30d
Description
I have a repository with a quite a lot of dependencies (node_modules size about 225mb with 625 folders). Code is written in TypeScript.
I've noticed that ncc build seems to run quite slow (around 3 to 5 seconds per build). That didn't seem normal (most dependencies are for development, so they shouldn't be involved with the build process). I tried tweaking bunch of things to get an idea what's wrong, but nothing really seemed to have too much of an effect on the build time (removing imports of larger dependencies etc.)
I suspected TypeScript for a while, so I ended up checking how fast the now-examples/typescript build ran. It took about 0.5 to 1 second on my machine, so something was clearly different with that one.
Did a small experiment with adding a bunch of random dependencies to the now-examples/typescript repository, and seeing if that had any effect on the build time. It seems to slow the building a lot, even without the code importing any of those dependencies.
Here's how to reproduce this:
yarn global add @zeit/ncc
git clone https://github.com/zeit/now-examples
cd now-examples/typescript
yarn
ncc build --watch ./src/frontend/index.ts
With this, the build times are as follows:
ncc: Version 0.18.2
ncc: Compiling file index.js
ncc: Using typescript@3.4.5 (local user-provided)
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [2318ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [1247ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [731ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [600ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [596ms] - ncc 0.18.2
Watching for changes...
Adding some random node modules after this:
yarn add lodash request chalk commander async react debug yargs minimist colors prop-types mkdirp react-dom fs-extra moment vue axios postgraphile semver babel babel-loader eslint
Ending up with node_modules of about 120mb and 432 folders. Then running the same build command as before, I end up with the following build times:
ncc: Version 0.18.2
ncc: Compiling file index.js
ncc: Using typescript@3.4.5 (local user-provided)
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [3119ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [2723ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [1852ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [1717ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [1598ms] - ncc 0.18.2
Watching for changes...
File change, rebuilding...
ncc: Module directory "/now-examples/typescript/node_modules/node-fetch/lib" attempted to require "encoding" but could not be resolved, assuming external.
6kB dist/style.css
48kB dist/index.js
54kB [1831ms] - ncc 0.18.2
Watching for changes...
Seems that build times are 2-3 times higher with just random stuff in the node_modules folder that isn't even used.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the timing difference in now-examples/typescript with ncc build --watch ./src/frontend/index.ts, first with the original dependencies and then after the listed packages are added. Investigate why unused node_modules folders affect the build, using the reported ncc 0.18.2 output as the baseline; done means unused dependencies no longer cause this slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, typescript
- Domain
- build-system, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100