[BUG] npm does not update any dependencies `package.json` has a `file:` dependency in a certain state

Note: this may overlap with #3057, but I'm opening this separately because the trigger I was able to reproduce was a file: dependency, which isn't involved in that issue.

Current Behavior:

Running npm install leaves dependencies in node_modules at outdated versions, despite an updated spec in package.json

Expected Behavior:

npm install should ensure any dependencies are updated to the versions specified in package.json

Steps To Reproduce:

I've created a minimal reproduction as a git repo here: https://github.com/cincodenada/npm-lockfile-error-example

There are two ways to reproduce: for the short way, clone the repo, which is set up with two subfolders:

• local-library is a npm project that has a dependency. The specific package doesn't seem to matter, but I couldn't reproduce the issue unless local-library had at least one dependency.
• dependent-project is a second project, which has two dependencies: file:../local-library and some other dependency. I used faker because that's what triggered it for us, but again, the specific package doesn't seem to matter. In the main branch this depends on faker@^5.5.3, in faker4 it is faker@^4.1.0

To reproduce, do the following:
(Note: As originally posted this involved npm install --no-save faker@4, but I added a branch to make clearer what the use case is)

cd dependent_project
# initialize node_modules with old version
git checkout faker4
npm install
# "pull" update and try to install updated packages
git checkout main
npm install

The output I get is as follows:

 in dependent-project/ on main with unpushed
› npm ls
dependent-project@1.0.0 /Users/joel/repro/lockfile-error/dependent-project
├── faker@4.1.0 invalid
└── local-library@1.0.0 -> /Users/joel/repro/lockfile-error/local-library

Showing that faker was not upgraded, despite being specified at ^5.5.3 in package.json. npm ls seems to know this is wrong as well.

In recent versions of NPM, even if I uninstall local-library, the issue persists, I assume because of the extraneous entry left over in the package.json as mentioned in #3141 (output snipped for brevity):

› npm uninstall local-library
› npm install
› npm ls
dependent-project@1.0.0 /Users/joel/repro/lockfile-error/dependent-project
└── faker@4.1.0 invalid

I tried removing the extraneous entry from package-lock.json manually, and after that an npm install results in an npm ls that seems to think faker has been upgraded, but the version in node_modules is actually unchanged.

Script from scratch

I also have created a script that creates the situation from scratch, to facilitate testing on different versions, and with different combinations of example packages. This is repro.sh in the root of the directory - it should be fairly self-explanatory, but in summary, it destroys and recreates the two project directories, installs the initial dependencies, ensures that the correct version was installed, installs the file: dependency, downgrades faker, runs npm install, and asserts that the dependency was upgraded, which should fail. If you comment out the set -e at the top, it will continue on to uninstall local-library and try downgrading and installing again, which for me fails in 7.11 but not in earlier versions - I've tested 7.0, 7.1, 7.2, and 7.3 so far, and all successfully upgrade after simply removing the link.

This script also takes arguments: repro.sh <package-name> <old-version> <new-version> will run through it, substituting the given package and versions, the base script is equivalent to repro.sh faker 4 5. This script also returns an error code if npm doesn't upgrade as it should, so should be suitable for git bisect or similar, if that's useful.

Other observations

One thing I noticed while poking around is that there's an error message when running npm install in the bad state:

npm verb shrinkwrap failed to load node_modules/.package-lock.json missing from lockfile: ../local-library/node_modules/ansi-styles

Which suggests that it's trying to find the dependencies of the symlinked dependency in its local package-lock.json and failing. I have noticed that sometimes our symlinked dependencies have second-level dependencies embedded in the top-level project's package-lock.json, and sometimes they don't, and I haven't determined why - we just upgraded from npm 6 to npm 7, so they may have just been left over from that. When I uninstalled and reinstalled all of our file: dependencies, none of them had the second-level dependencies embedded in the top-level project any more. This reduced our package-lock.json files by some 150k lines all told, so if we can stay in this state that would be great! I am unsure if this is related to the issue above or not.

I haven't attempted to reproduce a pacakge-lock.json with the second-level dependencies embedded, so I am unsure if this issue occurs in that situation or not.

Debug logs

I've also attached three debug logs from npm@7.11.1:
before installing the file: dep
after uninstalling but with the leftover extraneous entry
after installing

Environment:

OS: Mac OS Big Sur 11.2.3
Node: 12.22
npm: 7.11.1 (also reproduced with 7.0, 7.1, 7.2)

Start with the linked minimal reproduction and its root-level repro.sh, then run the checkout and npm install sequence described for dependent-project with npm 7.11.1. Compare npm ls, package-lock.json, and node_modules after switching from faker4 to main; done means the installed dependency matches the updated package.json version, including when the file: dependency is present.