[RRFC] npm install --from-lockfile

Motivation ("The Why")

Based on the discussion on npm/cli#564, npm/feedback/discussions/340 and a few other issues.

The purpose of this proposal is to offer a middle ground between npm install and npm ci, providing the speed benefits of partial installs (npm install with an existing node_modules folder) alongside (some of) the deterministic guarantees of npm ci (exact package numbers and dependency numbers from package-lock.json).

Example

1. At my current workplace, we run continuous deployment on a server for testing purposes. We have a large dependency tree with multiple dependencies that require node-gyp compilation.

   We do not use npm ci; the clean install process takes longer than we would like. We would like to push changes and very quickly get results back from the server. Additionally, because we are not building production builds, we are not concerned with doing a clean, completely deterministic install every time. We don't need that level of accuracy for a test server.

   However, we are interested in forcing our test server to respect package-lock.json. We have run into multiple failed builds that were the result of minor/bugfix package updates that automatically happened during a push. So ideally, we would be able to keep the speed of npm install, while respecting package-lock.json during the install.

2. We are onboarding a contractor to look over the codebase, or we have an Open Source project that we are writing compile instructions for. The contractor/contributor should use npm ci after they pull our repo, so we know they're getting all of the correct versions. However, even when writing documentation about how to download or install a repo, a contractor/contributor might forget to run npm ci. They might not even know the command exists. In the case of a contributor, they might turn around a file an issue or complain that the project isn't compiling.

   We'd like to be able to specify that npm install should use --from-lockfile in our project's .npmrc, which would completely mitigate the above issue.

3. One way we might decide to approach the above situations is by specifying exact versions of each package in our package.json. However, this would block us from being able to use wildcards and fuzzy package numbers to help with guiding the upgrading of packages. Additionally, contributors would need to remember to only install exact versions or to update package.json with exact numbers whenever they install a new package. In my experience, they often forget to do so.

   We'd like to be able to get the general behavior of using exact package numbers in our package.json without losing all of the benefits of fuzzy matching during package upgrades, and without complicating package installation.

How

Current Behaviour

There are two basic ways to install packages in NPM: npm install and npm ci:

A) npm install reads packages from package.json, which allows packages to be updated behind the scenes.
B) npm ci reads from package-lock.json, which allows much greater confidence that packages will not change behind someone else's back.
C) npm install allows partial installs: if a package is already up to date in node_modules, it will not be re-downloaded/compiled.
D) npm ci deletes the node_modules folder every run, forcing a clean install. This means every package will run its pre-install scripts, get copied over to node_modules, etc...

This comment on the original issue goes into a bit more detail, but the basic gist is that a full clean install is too slow for many non-production settings as soon as large packages with complicated pre/post-install scripts come into play. However, allowing non-production builds to grab the latest versions of dependencies is not reliable enough behavior for most environments.

Desired Behaviour

npm install --from-lockfile behaves according to B) and C) in "Current Behavior" above, while avoiding behavior A) and D).

• from-lockfile reads from package-lock.json or another shrinkwrap for exact package numbers.
• from-lockfile does not remove node_modules.
• from-lockfile is ignored when installing new packages.
• from-lockfile will error out if a package-lock.json or other shrinkwrap is not present.
• npm install --from-lockfile is not a clean install. If an existing installed module is corrupted or edited beyond the point where a normal npm install would catch it, it will not be cleared out and reinstalled. It does not offer the same deterministic guarantees that npm ci offers.

This specific solution/behavior is loosely based off of @jdussouillez's comment and @DanielRuf's comment that both encouraged adding a new flags to npm install instead of npm ci.

It deviates from their offered solutions in that it does not go all the way and add a --clean option to npm install, nor does it try to alias npm ci to npm install --from-lockfile --clean. I think those are reasonable solutions, but I'm just trying to make the smallest change possible that will make all of the desired behaviors possible without messing with npm ci at all.

I think it's a much safer, smaller, more predictable change to add a single flag to npm install rather than to try and mess with existing behavior in a way that would possibly be a breaking change.

References

I have a basic implementation in progress at danShumway/cli/commit/bc8ef2.... Assuming no other changes, I still need to add a few more unit tests and I still need to add documentation. But of course, that's all subject to whether or not the implementation/proposal itself still requires work.

From my very basic testing, this seems to be working and comes with a pretty substantial performance increase in the scenarios I describe above.

Review the proposed implementation at the linked danShumway/cli commit first, then inspect the npm CLI install behavior it changes. Add the remaining unit tests and documentation mentioned in the issue, and verify the requested lockfile-based partial-install behavior and its stated edge cases.