[BUG] overriding a dependency's dependency with a scoped package, within the dependency, resolves it to the wrong parent path

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

I install a library foo as a dependency of my project. That library has a dependency, bar. I configure an override of bar (within foo) with my own fork, @myorgscope/bar, by putting this in package.json:

  "overrides": {
    "foo": {
      "bar": "@myorgscope/bar"
    }
  }

and then I run npm i.

I now have this in my package-lock.json:

$ jq '.packages["node_modules/foo/node_modules/bar"]' package-lock.json
{
  "resolved": "node_modules/foo/@myorgscope/bar",
  "link": true
}

and, correspondingly, this symlink in my filesystem:

$ readlink node_modules/foo/node_modules/bar
../@myorgscope/bar

This symlink, of course, resolves to node_modules/foo/@myorgscope/bar which does not exist:

ls node_modules/foo/node_modules/bar/
ls: node_modules/foo/node_modules/bar/: No such file or directory

and sure enough, requiring foo will fail to require its dependency bar:

node -e 'require("foo")'
node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'bar'
Require stack:
[...]`

Expected Behavior

Instead, I should have this in my package-lock.json:

$ jq '.packages["node_modules/foo/node_modules/bar"]' package-lock.json
{
  "resolved": "node_modules/@myorgscope/bar",
  "link": true
}

(the difference being that the foo/ path element goes away)
and, correspondingly, this symlink in my filesystem:

$ readlink node_modules/foo/node_modules/bar
../../@myorgscope/bar

This symlink would resolve to node_modules/@myorgscope/bar which does exist:

ls node_modules/foo/node_modules/bar/
[list of all the files at the root level of the `@myorgscope/bar` library]

and requiring foo would then work.

Indeed, if I manually edit package-lock.json to remove the foo/ path element in that problemated "resolved" attribute, and run npm ci, the symlink is then correct and requiring foo does work.

Steps To Reproduce

1. Publish a scoped fork of pgsql-ast-parser

git clone git@github.com:oguimbal/pgsql-ast-parser.git
cd pgsql-ast-parser
npm i
npm run build
cd lib
export NPM_SCOPE=$USER # or whatever your logged-in npm username is (or some npmjs.com scope you can publish into for testing
gsed -i 's#"pgsql-ast-parser"#"@'"$NPM_SCOPE"'/pgsql-ast-parser"#' package.json
gsed -i 's#"Yet another simple Postgres SQL parser/modifier"#"Dummy fork just to illustrate an npm bug"#' package.json
npm publish --access public

2. Set up a fresh npm project with a dependency on pg-mem and your fork of its dependency:

mkdir npm-cli-fork-override-symlink-bug-repro
cd npm-cli-fork-override-symlink-bug-repro
npm init -y
npm i --save pg-mem @"$NPM_SCOPE"/pgsql-ast-parser

3. Edit package.json to insert this override:

  "overrides": {
    "pg-mem": {
      "pgsql-ast-parser": "@gthb/pgsql-ast-parser@^10.5.2"
    }
  }

(note that you must include the version spec like in your direct dependency, else the override silently fails to be applied, and you can't use the dollar-prefixed $@gthb/pgsql-ast-parser way to express that; apparently it does not work with scoped packages ... I think those are separate bugs)

5. Apply the override:

npm i

7. Observe the buggy results ...

... that the symlink was created incorrectly:

$ ls -l node_modules/pg-mem/node_modules
total 0
lrwxr-xr-x  1 gthb  staff  25 Jun 24 10:52 pgsql-ast-parser -> ../@gthb/pgsql-ast-parser

... and that the resolved path is incorrect in package-lock.json:

$ jq '.packages["node_modules/pg-mem/node_modules/pgsql-ast-parser"]' package-lock.json
{
  "resolved": "node_modules/pg-mem/@gthb/pgsql-ast-parser",
  "link": true
}

... and that pg-mem fails to load:

$ node -e 'require("pg-mem")'
node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'pgsql-ast-parser'

That reproduces the error.

Now to demonstrate a manual workaround: edit package-lock.json to change the line:

"resolved": "node_modules/pg-mem/@gthb/pgsql-ast-parser@^10.5.2",

to:

"resolved": "node_modules/@gthb/pgsql-ast-parser@^10.5.2",

and run npm ci ... and see that fail!

$ npm ci
npm ERR! code EUSAGE
npm ERR!
npm ERR! The `npm ci` command can only install with an existing package-lock.json or
npm ERR! npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm@5 or
npm ERR! later to generate a package-lock.json file, then try again.
[...]

... and so, edit that line again to remove the version specifier:

"resolved": "node_modules/@gthb/pgsql-ast-parser",

and now run npm ci. It should succeed, after which the symlink points to the right place:

$ ls -l node_modules/pg-mem/node_modules
total 0
lrwxr-xr-x  1 gthb  staff  28 Jun 24 11:16 pgsql-ast-parser -> ../../@gthb/pgsql-ast-parser

and importing pg-mem works:

$ node -e 'const db = require("pg-mem").newDb(); console.log(db.public.one("select 3.14"))'
{ column: 3.14, [Symbol(_id)]: 'dual_0', [Symbol()]: Symbol() }

Environment

• npm: 8.13.1
• Node.js: v16.15.1
• OS Name: macOS Monterey Version 12.4
• System Model Name: Macbook Pro
• npm config:

; "user" config from /Users/gthb/.npmrc

@grid-is:registry = "https://registry.npmjs.org/"
//registry.npmjs.org/:_authToken = (protected)

; node bin location = /Users/gthb/.fnm/node-versions/v16.15.1/installation/bin/node
; node version = v16.15.1
; npm local prefix = /Users/gthb/git/npm-cli-fork-override-symlink-bug-repro
; npm version = 8.13.1
; cwd = /Users/gthb/git/npm-cli-fork-override-symlink-bug-repro
; HOME = /Users/gthb
; Run `npm config ls -l` to show all defaults.

Start with the package.json override and the package-lock.json entry produced by npm i, then reproduce the scoped-package case using the commands and pg-mem example in the issue. Compare the lockfile's resolved path and the node_modules symlink, and verify the result with npm ci and require("pg-mem"). Done means the resolved path omits the dependency parent directory, the symlink targets the installed scoped package, and pg-mem loads successfully.