[BUG] npm ls throws invalid package error for optional dependency of a different version found in dependency tree

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Consider this package json and package lock json setup :https://github.com/ashlymat/npmDummy

• Clone the above repo
• Run npm ci
• Run npm ls --json --long --all with or without --omit=optional

I was able to reproduce with 8.19.1 npm locally and with docker image node:18 which has npm 9.5.1

Error seen:

npm ERR! code ELSPROBLEMS
npm ERR! invalid: commander@2.9.0 /Users/I560834/Dev/others/npmDummy/node_modules/commander
{
  "error": {
    "code": "ELSPROBLEMS",
    "summary": "invalid: commander@2.9.0 /Users/I560834/Dev/others/npmDummy/node_modules/commander",
    "detail": ""
  }
}

In the above repo, .npmrc has a setting of omit=optional

when npm ci is run , it respects this setting and does not install optional dependency for commander package
and hence there is nothing installed in node_modules/z-schema/node_modules/commander folder.

But when npm ls --json --long --all is run , somehow this optional setting is not respected and it complains about another version of the same commander package being installed in the parent node_modules folder.
This version in the parent folder is a dependecy from another package @sap/approuter itself.

The problem goes away if we remove omit=optional setting in .npmrc. This is because it then has the correct version installed in node_modules/z-schema/node_modules/commander folder.

The problem also doesn't exist if there was no other version of commander package at all in the node_modules folder.

Summarizing it,

npm ls --all seems to have an issue in omitting optional dependencies when another version of the dependency is present in node_modules.

Expected Behavior

It seems like npm ls should have a check to identify if the optional dependency of a different version in the dependency tree - is required by some other package in the tree, before throwing an error.

Steps To Reproduce

Have been described above

Environment

• npm: 8.9.1 and 9.5.1
• Node.js: node: 18 docker image
• OS Name:
• System Model Name:
• npm config:

; copy and paste output from `npm config ls` here

Clone the linked npmDummy repository, run npm ci, then compare npm ls --json --long --all with and without --omit=optional. Trace the npm CLI dependency-tree inspection that reports commander@2.9.0 as invalid when the optional nested dependency is omitted. Done means the reproduced setup no longer reports this optional dependency as invalid while preserving validation of genuinely required dependencies.