Node >= 16.18.0 breaks npm ci

Version

16.18.0, 16.19.0, 18.3.0

Platform

Linux ip-10-94-32-11 5.15.0-1026-aws #30-Ubuntu SMP Wed Nov 23 14:15:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

http

What steps will reproduce the bug?

Execute npm ci

How often does it reproduce? Is there a required condition?

I am able to reproduce the issue every single time.

What is the expected behavior?

npm ci should execute successfully.

What do you see instead?

$ npm ci
npm ERR! code EPIPE
npm ERR! syscall write
npm ERR! errno -32
npm ERR! write EPIPE

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2023-01-12T18_17_28_239Z-debug-0.log

Additional information

I originally left a comment on closed issue #44998. In fear that it would never be seen, I've opened a new issue here.

I am currently experiencing this issue on any version of node after 16.18.0. I have tried the latest versions from both the 16 and 18 release channels.

Using git bisect and building node from source, I have narrowed down the problematic commit to

5f406a3a53: http: reuse socket only when it is drained (3 months ago) <ywave620>. (associated pull request https://github.com/nodejs/node/pull/43902)

When I run npm ci on any branch which includes this commit, I encounter this EPIPE error. If I revert the commit and rebuild node, I can see from the logs that the EPIPE error is still emitted, but it is handled properly.

Unfortunately, I'm really not sure what exactly it is about my environment that is causing this EPIPE error, as it doesn't seem like it's affecting very many users at all. I experience the issue running from AWS EC2 or ECS connected to my company's private network using a Transit Gateway. So, there must be something special about my network which triggers this issue.

In the meantime, the best I can currently do is attach a set of detailed logs.

For each of these sets of logs, I ran DEBUG='*' NODE_DEBUG='*' npm ci:

• v16.17.1 (Working version)
• v16.18.0 (First released broken version)
• v16.19.0 (Latest release on v16 LTS branch)

To quickly find the error in these logs, just search for the string 'EPIPE'. In summary, it looks like in v16.17.1, EPIPE is emitted but handled; in later versions EPIPE is emitted but is unhandled.

Furthermore, in reading through the code on my own, I am seeing a significant quantity of comments labeled TODO(ronag): that mention edge cases for error handling. I think may be the unlucky soul who is hitting these edge cases. I think, in particular, this TODO is the one that is doing me in (judging from my log output, as well as some debug statements I've added during my own test runs).

Please let me know if there's any further information I can provide for troubleshooting. As it seems like it won't be possible for any Node developers to reproduce this issue, I am more than happy to rebuild and test Node myself with any recommended changes to see if they fix my issue (I can even add additional debug statements and rerun npm ci if that helps in any way).

Thank you so much for any help!

Start with lib/_http_agent.js, especially the linked TODO(ronag), and inspect commit 5f406a3a53 and its pull request. Reproduce with DEBUG='' NODE_DEBUG='' npm ci and compare the supplied v16.17.1, v16.18.0, and v16.19.0 logs around EPIPE handling. Done means npm ci succeeds in the affected environment and the EPIPE is handled as in the working version.