npm / npm/cli

[BUG] `npm i` fails for a package that exists in uplink, but, does not exist in private registry

Open
#5,045 1 comment 12 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Needs Triage Release 8.x
Dominant language
JavaScript
Stars
10.1k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
19

Description

Is there an existing issue for this?
  • I have searched the existing issues
This issue exists in the latest npm version
  • I am using the latest npm
Current Behavior

We have private registry.
Our private registry has these 2 endpoints:

  1. GET /package/:artefactName/:version: This returns the metadata. If the package is not in the private repo, It queries the https://registry.npmjs.org/ for metadata and returns the response
  2. GET /package/:artefactName/-/:bundleName: This returns the tarball only if available in the private data store. Otherwise, throws error

We have added .npmrc with the following configs to one of our npm projects:

registry=http://localhost:8081/api/8.3.0/package
# package-lock=false
# save-prefix=
Issue:
  • User executes npm install sprintf-js command
  • npm requests the package metadata from the private registry
  • server responds with the correct metadata
  • metadata file for sprintf-js package: sprintf-js.metadata.json.zip
  • the tarball url for version 1.1.2 points to https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.2.tgz
  • But, npm requests the private registry for the tarball: GET http://localhost:8081/api/8.3.0/package/sprintf-js/-/sprintf-js-1.1.2.tgz
  • This does not exist in the private registry data source and hence, it fails

See below logs for this issue where npm i fails.

Here is the verbose log of `npm i --logLevel verbose` 
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
npm verb cli C:\Program Files\nodejs\node.exe C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js
npm info using npm@8.12.2
npm info using node@v16.15.1
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\npmrc Completed in 3ms
npm timing config:load:builtin Completed in 3ms
npm timing config:load:cli Completed in 2ms
npm timing config:load:env Completed in 1ms
npm timing config:load:file:D:\npm-i-test\.npmrc Completed in 1ms
npm timing config:load:project Completed in 3ms
npm timing config:load:file:C:\Users\Sankarshan\.npmrc Completed in 0ms
npm timing config:load:user Completed in 1ms
npm timing config:load:file:C:\Users\Sankarshan\AppData\Roaming\npm\etc\npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 2ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 15ms
npm timing npm:load:configload Completed in 16ms
npm timing npm:load:mkdirpcache Completed in 2ms
npm timing npm:load:mkdirplogs Completed in 0ms
npm verb title npm i
npm verb argv "i" "--loglevel" "verbose"
npm timing npm:load:setTitle Completed in 2ms
npm timing config:load:flatten Completed in 3ms
npm timing npm:load:display Completed in 14ms
npm verb logfile logs-max:10 dir:C:\Users\Sankarshan\AppData\Local\npm-cache\_logs
npm verb logfile C:\Users\Sankarshan\AppData\Local\npm-cache\_logs\2022-06-20T13_49_10_934Z-debug-0.log
npm timing npm:load:logFile Completed in 19ms
npm timing npm:load:timers Completed in 0ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load Completed in 56ms
npm timing arborist:ctor Completed in 0ms
npm timing arborist:ctor Completed in 0ms
npm timing idealTree:init Completed in 18ms
npm timing idealTree:userRequests Completed in 0ms
npm http fetch GET 200 http://localhost:8081/api/8.3.0/package/sprintf-js 408ms (cache miss)
npm timing idealTree:#root Completed in 435ms
npm timing idealTree:node_modules/sprintf-js Completed in 0ms
npm timing idealTree:buildDeps Completed in 440ms
npm timing idealTree:fixDepFlags Completed in 1ms
npm timing idealTree Completed in 465ms
npm timing reify:loadTrees Completed in 467ms
npm timing reify:diffTrees Completed in 2ms
npm timing reify:retireShallow Completed in 0ms
npm timing reify:createSparse Completed in 6ms
npm timing reify:loadBundles Completed in 0ms
npm http fetch POST 404 http://localhost:8081/api/8.3.0/package/-/npm/v1/security/advisories/bulk 26ms
npm http fetch POST 404 http://localhost:8081/api/8.3.0/package/-/npm/v1/security/audits/quick 13ms
npm verb audit error HttpErrorGeneral: 404 Not Found - POST http://localhost:8081/api/8.3.0/package/-/npm/v1/security/audits/quick
npm verb audit error     at C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\npm-registry-fetch\lib\check-response.js:93:15
npm verb audit error     at processTicksAndRejections (node:internal/process/task_queues:96:5)
npm verb audit error     at async Map.[getReport] (C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\audit-report.js:333:21)
npm verb audit error     at async Map.run (C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\audit-report.js:106:19)
npm verb audit error  HttpErrorGeneral: 404 Not Found - POST http://localhost:8081/api/8.3.0/package/-/npm/v1/security/audits/quick
npm verb audit error     at C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\npm-registry-fetch\lib\check-response.js:93:15
npm verb audit error     at processTicksAndRejections (node:internal/process/task_queues:96:5)
npm verb audit error     at async Map.[getReport] (C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\audit-report.js:333:21)
npm verb audit error     at async Map.run (C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\audit-report.js:106:19) {
npm verb audit error   headers: [Object: null prototype] {
npm verb audit error     'x-dns-prefetch-control': [ 'off' ],
npm verb audit error     'x-frame-options': [ 'SAMEORIGIN' ],
npm verb audit error     'strict-transport-security': [ 'max-age=15552000; includeSubDomains' ],
npm verb audit error     'x-download-options': [ 'noopen' ],
npm verb audit error     'x-content-type-options': [ 'nosniff' ],
npm verb audit error     'x-xss-protection': [ '1; mode=block' ],
npm verb audit error     'access-control-allow-origin': [ '*' ],
npm verb audit error     'content-security-policy': [ "default-src 'self'" ],
npm verb audit error     'content-type': [ 'text/html; charset=utf-8' ],
npm verb audit error     'content-length': [ '188' ],
npm verb audit error     date: [ 'Mon, 20 Jun 2022 13:49:11 GMT' ],
npm verb audit error     connection: [ 'keep-alive' ],
npm verb audit error     'keep-alive': [ 'timeout=5' ],
npm verb audit error     'x-fetch-attempts': [ '1' ]
npm verb audit error   },
npm verb audit error   statusCode: 404,
npm verb audit error   code: 'E404',
npm verb audit error   method: 'POST',
npm verb audit error   uri: 'http://localhost:8081/api/8.3.0/package/-/npm/v1/security/audits/quick',
npm verb audit error   body: <Buffer 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 3e 0a 3c 68 74 6d 6c 20 6c 61 6e 67 3d 22 65 6e 22 3e 0a 3c 68 65 61 64 3e 0a 3c 6d 65 74 61 20 63 68 61 72 ... 138 more bytes>,
npm verb audit error   pkgid: 'quick'
npm verb audit error }
npm timing auditReport:getReport Completed in 91ms
npm timing reify:audit Completed in 92ms
npm http fetch GET 500 http://localhost:8081/api/8.3.0/package/sprintf-js/-/sprintf-js-1.1.2.tgz 70274ms attempt #3 (cache skip)
npm timing reify:rollback:createSparse Completed in 9ms
npm timing reify:rollback:retireShallow Completed in 0ms
npm timing command:i Completed in 70835ms
npm verb stack HttpErrorGeneral: 500 Internal Server Error - GET http://localhost:8081/api/8.3.0/package/sprintf-js/-/sprintf-js-1.1.2.tgz
npm verb stack     at C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\node_modules\npm-registry-fetch\lib\check-response.js:93:15
npm verb stack     at processTicksAndRejections (node:internal/process/task_queues:96:5)
npm verb statusCode 500
npm verb pkgid sprintf-js@http://localhost:8081/api/8.3.0/package/sprintf-js/-/sprintf-js-1.1.2.tgz
npm verb cwd D:\npm-i-test
npm verb Windows_NT 10.0.19044
npm verb node v16.15.1
npm verb npm  v8.12.2
npm ERR! code E500
npm ERR! 500 Internal Server Error - GET http://localhost:8081/api/8.3.0/package/sprintf-js/-/sprintf-js-1.1.2.tgz
npm verb exit 1
npm timing npm Completed in 71138ms
npm verb unfinished npm timer reify 1655732951223
npm verb unfinished npm timer reify:unpack 1655732951730
npm verb unfinished npm timer reifyNode:node_modules/sprintf-js 1655732951731
npm verb code 1

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Sankarshan\AppData\Local\npm-cache\_logs\2022-06-20T13_49_10_934Z-debug-0.log

Compare this to the yarn install --verbose log

  • It queries the private registry for the package metadata: Performing "GET" request to "http://localhost:8081/api/8.3.0/package/sprintf-js"
  • From the metadata, it figures out that it has to hit the public npm registry
  • Fetches the package tarball from the public registry: Performing "GET" request to "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.2.tgz".
  • Installation succeeds

Note: This installation succeeds if done in npm v6.x.x

Here is the verbose log of `yarn install --verbose` yarn version: 1.22.19 
verbose 0.2059042 Checking for configuration file "D:\\npm-i-test\\.npmrc".
verbose 0.2067221 Found configuration file "D:\\npm-i-test\\.npmrc".
verbose 0.2078709 Checking for configuration file "C:\\Users\\Sankarshan\\.npmrc".
verbose 0.2084984 Found configuration file "C:\\Users\\Sankarshan\\.npmrc".
verbose 0.2095982 Checking for configuration file "C:\\Program Files\\nodejs\\etc\\npmrc".
verbose 0.2103793 Checking for configuration file "D:\\npm-i-test\\.npmrc".
verbose 0.2117668 Found configuration file "D:\\npm-i-test\\.npmrc".
verbose 0.2157005 Checking for configuration file "D:\\npm-i-test\\.yarnrc".
verbose 0.2165246 Found configuration file "D:\\npm-i-test\\.yarnrc".
verbose 0.2176542 Checking for configuration file "C:\\Users\\Sankarshan\\.yarnrc".
verbose 0.2184824 Found configuration file "C:\\Users\\Sankarshan\\.yarnrc".
verbose 0.2194361 Checking for configuration file "C:\\Program Files\\nodejs\\etc\\yarnrc".
verbose 0.2199986 Checking for configuration file "D:\\npm-i-test\\.yarnrc".
verbose 0.2204214 Found configuration file "D:\\npm-i-test\\.yarnrc".
verbose 0.2254752 current time: 2022-06-20T12:53:06.420Z
info No lockfile found.
[1/4] Resolving packages...
verbose 0.2727569 Performing "GET" request to "http://localhost:8081/api/8.3.0/package/sprintf-js".
verbose 0.4453627 Request "http://localhost:8081/api/8.3.0/package/sprintf-js" finished with status code 200.
[2/4] Fetching packages...
verbose 0.4788914 Performing "GET" request to "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.2.tgz".
[3/4] Linking dependencies...
verbose 0.7609122 Creating directory "D:\\npm-i-test\\node_modules\\sprintf-js".
verbose 0.7661962 Creating directory "D:\\npm-i-test\\node_modules\\sprintf-js\\dist".
verbose 0.7673571 Creating directory "D:\\npm-i-test\\node_modules\\sprintf-js\\src".
verbose 0.7717975 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\CONTRIBUTORS.md" to "D:\\npm-i-test\\node_modules\\sprintf-js\\CONTRIBUTORS.md".
verbose 0.7735682 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\CHANGELOG.md" to "D:\\npm-i-test\\node_modules\\sprintf-js\\CHANGELOG.md".
verbose 0.774956 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\LICENSE" to "D:\\npm-i-test\\node_modules\\sprintf-js\\LICENSE".
verbose 0.7759356 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\package.json" to "D:\\npm-i-test\\node_modules\\sprintf-js\\package.json".
verbose 0.7769853 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\README.md" to "D:\\npm-i-test\\node_modules\\sprintf-js\\README.md".
verbose 0.7780286 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\dist\\.gitattributes" to "D:\\npm-i-test\\node_modules\\sprintf-js\\dist\\.gitattributes".
verbose 0.7789373 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\dist\\angular-sprintf.min.js" to "D:\\npm-i-test\\node_modules\\sprintf-js\\dist\\angular-sprintf.min.js".
verbose 0.7799692 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\dist\\angular-sprintf.min.js.map" to "D:\\npm-i-test\\node_modules\\sprintf-js\\dist\\angular-sprintf.min.js.map".
verbose 0.7810647 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\dist\\sprintf.min.js" to "D:\\npm-i-test\\node_modules\\sprintf-js\\dist\\sprintf.min.js".
verbose 0.7823663 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\dist\\sprintf.min.js.map" to "D:\\npm-i-test\\node_modules\\sprintf-js\\dist\\sprintf.min.js.map".
verbose 0.783775 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\src\\angular-sprintf.js" to "D:\\npm-i-test\\node_modules\\sprintf-js\\src\\angular-sprintf.js".
verbose 0.7848578 Copying "C:\\Users\\Sankarshan\\AppData\\Local\\Yarn\\Cache\\v6\\npm-sprintf-js-1.1.2-da1765262bf8c0f571749f2ad6c26300207ae673-integrity\\node_modules\\sprintf-js\\src\\sprintf.js" to "D:\\npm-i-test\\node_modules\\sprintf-js\\src\\sprintf.js".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 0.64s.
Expected Behavior

After getting the metadata from the private registry, npm should resolve the package tarball from the url in the metadata. In this case, public npm registry

Steps To Reproduce
  1. Create a server that implements the below 2 end points:
  • GET /package/:artefactName/:version: This should return the metadata. If the package is not in the private repo, It should query the https://registry.npmjs.org/ for metadata and return the response
  • GET /package/:artefactName/-/:bundleName: This should return the tarball only if available in the private data store. Otherwise, throw error
  1. Create a folder npm-i-test
  2. cd to that folder and npm init -y
  3. create .npmrc with the following configs at the root of the project
registry=http://localhost:8081/api/8.3.0/package
# package-lock=false
# save-prefix=
  1. Clear the npm cache
  2. Install a publicly published package, for example, sprintf-js: npm i sprintf-js --logLevel verbose

Now, the npm i will get stuck for some time. After that it fails with the error similar to the above log

Environment
  • npm: 8.12.2
  • Node.js: 16.15.1
  • OS Name: Windows 10
  • System Model Name: Lenovo Ideadpad L340
  • npm config:
; "builtin" config from C:\Users\Sankarshan\AppData\Roaming\npm\node_modules\npm\npmrc

prefix = "C:\\Users\\Sankarshan\\AppData\\Roaming\\npm"

; "user" config from C:\Users\Sankarshan\.npmrc

//registry.npmjs.org/:_authToken = (protected)

; "project" config from D:\npm-i-test\.npmrc

registry = "http://localhost:8081/api/8.3.0/package"

; node bin location = C:\Program Files\nodejs\node.exe
; node version = v16.15.1
; npm local prefix = D:\npm-i-test
; npm version = 8.12.2
; cwd = D:\npm-i-test
; HOME = C:\Users\Sankarshan
; Run `npm config ls -l` to show all defaults.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with npm v8's tarball-fetch path, using the npm-registry-fetch and @npmcli/arborist reify flow shown in the log; compare it with npm v6 and Yarn. Reproduce npm i sprintf-js against the described private registry, and consider the issue done when metadata can come from the private registry while the public tarball URL is fetched successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
cli, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.