[BUG] unrecoverable "invalid or damaged lockfile" when dependencies have 'file:' subdependencies

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Could be related to #4664, but this one is slightly different: it seems to be caused by 'file:' subdependencies (which are probably a bad idea, but we already have packages with 'file:' dependencies in the registry), and it is unrecoverable; npm install produces "invalid" package-lock.json which is not accepted by npm ci from the same npm version.

Expected Behavior

Best case: npm ci should install dependencies using existing package-lock.json without any errors.
In any way, I expect npm install to produce correct package-lock.json; I expect that after npm install installed dependencies successfully and without errors, npm ci should not fail.

Steps To Reproduce

localhost:~/projects/irrelevant$ npm view @privatescope/offending-package@offending.version

@privatescope/offending-package@offending.version | Proprietary | deps: 12 | versions: 224
...description...

dist
.tarball: https://registry.npmjs.org/@privatescope/offending-package/-/offending-package-offending.version.tgz
.shasum: (irrelevant)
.integrity: sha512-(irrelevant)
.unpackedSize: (irrelevant)

dependencies:
...
@privatescope/local-package: file:../local-package
...

maintainers:
...

dist-tags:
...

published 3 months ago by (irrelevant)
localhost:~/projects/irrelevant$ npm ci
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: @privatescope/local-package@ from lock file
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/inga/.npm/_logs/2022-09-05T11_09_01_247Z-debug-0.log
localhost:~/projects/irrelevant$ cat /home/inga/.npm/_logs/2022-09-05T11_09_01_247Z-debug-0.log
(...)
39 silly idealTree buildDeps
40 timing idealTree:#root Completed in 0ms
41 silly placeDep ROOT @privatescope/local-package@ OK for: @private-scope/offending-package@offending.version want: file:../local-package
42 timing idealTree:node_modules/@privatescope/offending-package Completed in 10ms
43 timing idealTree:buildDeps Completed in 12ms
44 timing idealTree:fixDepFlags Completed in 4ms
45 timing idealTree Completed in 32ms
46 timing command:ci Completed in 258ms
47 verbose stack Error: `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file wit
h `npm install` before continuing.
47 verbose stack
47 verbose stack Missing: @privatescope/local-package@ from lock file
47 verbose stack
47 verbose stack     at CI.exec (/usr/lib/node_modules/npm/lib/commands/ci.js:73:13)
47 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli.js:78:5)
48 verbose cwd /home/inga/projects/irrelevant
49 verbose Linux 5.15.64-0-lts
50 verbose node v18.8.0
51 verbose npm  v8.10.0
52 error `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install`
 before continuing.
52 error
52 error Missing: @privatescope/local-package@ from lock file
52 error
(...)
localhost:~/projects/irrelevant$ grep -B 6 local-package package-lock.json 
        "node_modules/@privatescope/offending-package": {
            "version": "offending.version",
            "resolved": "https://registry.npmjs.org/@privatescope/offending-package/-/offending-package-offending.version.tgz",
            "integrity": "sha512-irrelevant",
            "dependencies": {
                "@privatescope/irrelevant": "^irrelevant.version",
                "@privatescope/local-package": "file:../local-package",
--
        "@privatescope/offending-package": {
            "version": "offending.version",
            "resolved": "https://registry.npmjs.org/@privatescope/offending-package/-/offending-package-offending.version.tgz",
            "integrity": "sha512-irrelevant",
            "requires": {
                "@privatescope/irrelevant": "^irrelevant.version",
                "@privatescope/local-package": "file:../local-package",
localhost:~/projects/irrelevant$ rm -rf node_modules/ && rm -rf package-lock.json 
localhost:~/projects/irrelevant$ npm install
npm WARN reify invalid or damaged lockfile detected
npm WARN reify please re-try this operation once it completes
npm WARN reify so that the damage can be corrected, or perform
npm WARN reify a fresh install with no lockfile if the problem persists.
npm WARN deprecated (a bunch of deprecated third-party packages follows)

added X packages, and audited Y packages in Zs

W packages are looking for funding
  run `npm fund` for details

N vulnerabilities (M low, L moderate, K high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
localhost:~/projects/irrelevant$ npm ci
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: @privatescope/local-package@ from lock file
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/inga/.npm/_logs/2022-09-05T11_11_05_188Z-debug-0.log
localhost:~/projects/irrelevant$ 

(Rerunning npm install again, preserving or removing package-lock.json, does not affect anything; npm ci still fails, it seems that there is no way to get a correct package-lock.json)

Environment

• npm: 8.18.0 (the issue is also present in 8.10.0, but not present in 8.1.3)
• Node.js: 18.8.0
• OS Name: Alpine Linux (edge)
• System Model Name: x86-64 PC
• npm config:

; "builtin" config from /usr/lib/node_modules/npm/npmrc

globalignorefile = "/etc/npmignore" 
prefix = "/usr/local" 
python = "/usr/bin/python3" 

; "user" config from /home/inga/.npmrc

//registry.npmjs.org/:_authToken = (protected) 

; node bin location = /usr/bin/node
; node version = v18.8.0
; npm local prefix = /home/inga/projects/(irrelevant)
; npm version = 8.10.0
; cwd = /home/inga/projects/(irrelevant, same as npm local prefix)
; HOME = /home/inga
; Run `npm config ls -l` to show all defaults.

The reported failure surfaces in npm/lib/commands/ci.js; start by comparing its lockfile validation with the idealTree and reify messages in the reproduction. Reproduce the file: subdependency case, then verify that npm install writes a lockfile accepted by npm ci without the missing-package error.