cloudflare / cloudflare/workerd
🐛 Infinite loop crash under WSL - failed: connect(): Cannot assign requested address; address = 127.0.0.1:60743
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
Hello,
I have a Remix application that uses `wrangler` to launch in dev mode, and it's been working for me for quite some time, until it no longer did.
# Environment
- Arch Linux under latest WSL2 on latest Windows 11
- Latest `wrangler`
- Node 21
# Bug Reproduction
The steps to reproduce the bug are:
1. `wrangler pages dev --compatibility-date=2023-06-21 ./public`
2. Load a page a few times with browser cache disabled (I'll explain why later)
3. Watch `workerd` enter an infinite loop of printing this error:
```
✘ [ERROR] workerd/io/io-context.c++:353: error: e = kj/async-io-unix.c++:1509: failed: connect(): Cannot assign requested address; address = 127.0.0.1:60743
stack:
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b95eee1
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b9560f1
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b955ea1
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b955c22
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b956a50
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b958dd0
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@48ca9f9
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b932130
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b937f00
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@55baa78
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@736f3ca
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8ec770
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8be930
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8fecf0
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@5a43838
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@5d24800
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@52c6f30
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@5a45356
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@5a4543e
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8b76b7
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8b766c
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8b76e3
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8ac096
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8c9c50
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8ae50c
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@b8aded8
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47c0257
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47d8da8
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47d870c
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47d86e8
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47d86be
/home/REDACTED/REDACTED/REDACTED/node_modules/.pnpm/@cloudflare+workerd-linux-64@1.20240405.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@47d85f3
```
> Note: The file path at the top of trace is sometimes `workerd/io/io-context.c++:353` and other times `workerd/jsg/util.c++:291`, not sure why.
**It's important to note that:**
- The error may start on first page load, or 10th. It doesn't after an exact step, it just occurs after using `wrangler` for "a while"
- It seems like disabling browser cache causes a lot more HTTP requests to go towards `wrangler` which makes the bug much more likely to trigger
- Once the bug starts `wrangler` is just virtually dead and spamming the console as CPU speed with this error
# Things I know about this bug
1. If I run `netstat -ntlp | grep ` I do see that a `node` process already reserved ``, which makes sense why `wrangler` can't acquire it, but then if I Ctrl-C `wrangler` itself that `node` process disappears, so basically `wrangler` itself was allocating it to begin with anyways.
2. If I run this same project on a cloud machine, this bug does not happen.
# Things I suspect may be relevant
1. I recently updated my WSL2 to latest. Now, I tried downgrading a bunch of versions, that did not fix the bug. I tried downgrading further then my WSL2 installation broke up so I re-upgraded to latest
# Debugging
I did the following:
1. Built `workerd` from source with `-c dbg` to preserve debug info in the output binary
2. Replaced `workerd` in my project's file tree with the one with debug info
3. Reproduced the bug but with `LLVM_SYMBOLIZER=$(which llvm-symbolizer)` and got the following:
```
kj::(anonymous namespace)::LowLevelAsyncIoProviderImpl::wrapConnectingSocketFd(int, sockaddr
const*, unsigned int, unsigned int) at external/capnp-cpp/src/kj/async-io-unix.c++:1509:11
kj::(anonymous namespace)::NetworkAddressImpl::connectImpl(kj::LowLevelAsyncIoProvider&,
kj::LowLevelAsyncIoProvider::NetworkFilter&, kj::ArrayPtr, bool)::'lambda'()::operator()() const at
external/capnp-cpp/src/kj/async-io-unix.c++:1648:25
decltype(reducePromiseType((kj::_::ReturnType_, bool)::'lambda'(), void>::Type*)(nullptr), false))
kj::evalNow, bool)::'lambda'()>(kj::(anonymous
namespace)::NetworkAddressImpl::connectImpl(kj::LowLevelAsyncIoProvider&,
kj::LowLevelAsyncIoProvider::NetworkFilter&, kj::ArrayPtr, bool)::'lambda'()&&)::'lambda'()::operator()() const at
external/capnp-cpp/src/kj/async-inl.h:1448:3
kj::Maybe
kj::runCatchingExceptions, bool)::'lambda'(), void>::Type*)(nullptr), false))
kj::evalNow, bool)::'lambda'()>(kj::(anonymous
namespace)::NetworkAddressImpl::connectImpl(kj::LowLevelAsyncIoProvider&,
kj::LowLevelAsyncIoProvider::NetworkFilter&, kj::ArrayPtr, bool)::'lambda'()&&)::'lambda'()>(kj::(anonymous
namespace)::NetworkAddressImpl::connectImpl(kj::LowLevelAsyncIoProvider&,
kj::LowLevelAsyncIoProvider::NetworkFilter&, kj::ArrayPtr, bool)::'lambda'()&&) at
bazel-out/k8-dbg/bin/external/capnp-cpp/src/kj/_virtual_includes/kj/kj/exception.h:366:5
kj::(anonymous namespace)::NetworkAddressImpl::connectImpl(kj::LowLevelAsyncIoProvider&,
kj::LowLevelAsyncIoProvider::NetworkFilter&, kj::ArrayPtr, bool)::'lambda'(kj::Own&&)::operator()(kj::Own&&) const at
external/capnp-cpp/src/kj/async-io-unix.c++:1652:0
kj::(anonymous
namespace)::NetworkAddressImpl::connect()::'lambda'(kj::AuthenticatedStream&&)::operator()(kj::AuthenticatedStream&&)
const at external/capnp-cpp/src/kj/async-io-unix.c++:1557:0
workerd::server::PromisedNetworkAddress::connect() (.resume) at
src/workerd/server/server.c++:440:17
kj::(anonymous
namespace)::PromisedAsyncIoStream::PromisedAsyncIoStream(kj::Promise>)::'lambda'(kj::Own)::operator()(kj::Own) const at
external/capnp-cpp/src/kj/async-io.c++:2376:0
kj::(anonymous namespace)::PromisedAsyncIoStream::write(void const*, unsigned
long)::'lambda'()::operator()() const at external/capnp-cpp/src/kj/async-io.c++:2422:0
kj::_::Coroutine::ForkedPromiseAwaiter::await_resume() at
bazel-out/k8-dbg/bin/external/capnp-cpp/src/kj/_virtual_includes/kj-async/kj/async-inl.h:2399:44
kj::(anonymous namespace)::HttpFixedLengthEntityWriter::write(void const*, unsigned long)
(.resume) at external/capnp-cpp/src/kj/compat/http.c++:2476:5
kj::(anonymous namespace)::AsyncPipe::BlockedPumpTo::write(void const*, unsigned
long)::'lambda'()::operator()() const at external/capnp-cpp/src/kj/async-io.c++:1250:0
kj::Canceler::AdapterImpl::AdapterImpl(kj::PromiseFulfiller&, kj::Canceler&,
kj::Promise)::$_0::operator()() const at external/capnp-cpp/src/kj/async.c++:272:0
kj::(anonymous namespace)::AsyncPipe::BlockedWrite::pumpTo(kj::AsyncOutputStream&, unsigned
long)::'lambda1'()::operator()() const at external/capnp-cpp/src/kj/async-io.c++:639:0
workerd::api::(anonymous
namespace)::BodyBufferInputStream::pumpTo(workerd::api::WritableStreamSink&, bool) (.resume) at
src/workerd/api/http.c++:598:7
workerd::api::ReadableStreamInternalController::pumpTo(workerd::jsg::Lock&,
kj::Own,
bool)::$_0::operator()(workerd::api::DeferredProxy) at
src/workerd/api/streams/internal.c++:2079:0
kj::Promise
workerd::IoContext::waitForDeferredProxy(kj::Promise>&&)::'lambda'(workerd::api::DeferredProxy)::operator()(workerd::api::DeferredProxy)
const at bazel-out/k8-dbg/bin/src/workerd/io/_virtual_includes/io/workerd/io/io-context.h:519:0
kj::Promise workerd::api::(anonymous namespace)::fetchImplNoOutputLock(workerd::jsg::Lock&,
workerd::jsg::Ref, workerd::jsg::Ref,
kj::Vector)::$_1::operator()>(kj::Promise,
workerd::jsg::Ref) const (.resume) at
src/workerd/api/http.c++:1818:11
kj::Promise workerd::api::(anonymous namespace)::fetchImplNoOutputLock(workerd::jsg::Lock&,
workerd::jsg::Ref, workerd::jsg::Ref,
kj::Vector)::$_1::operator()>(kj::Promise,
workerd::jsg::Ref) const (.resume) at
src/workerd/api/http.c++:1822:13
std::__1::coroutine_handle::resume[abi:ue170006]() const at
/usr/bin/../include/c++/v1/__coroutine/coroutine_handle.h:78:9
kj::_::CoroutineBase::fire() at external/capnp-cpp/src/kj/async.c++:3043:13
non-virtual thunk to kj::_::CoroutineBase::fire() at external/capnp-cpp/src/kj/async.c++:0:0
kj::EventLoop::turn() at external/capnp-cpp/src/kj/async.c++:1828:31
kj::_::waitImpl(kj::Own&&, kj::_::ExceptionOrValue&,
kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const at
external/capnp-cpp/src/kj/async.c++:1988:21
void kj::WaitScope::runOnStackPool&&, kj::_::ExceptionOrValue&, kj::WaitScope&,
kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own&&,
kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) at
external/capnp-cpp/src/kj/async.h:1320:7
kj::_::waitImpl(kj::Own&&, kj::_::ExceptionOrValue&,
kj::WaitScope&, kj::SourceLocation) at external/capnp-cpp/src/kj/async.c++:1985:17
kj::Promise::wait(kj::WaitScope&, kj::SourceLocation) at
bazel-out/k8-dbg/bin/external/capnp-cpp/src/kj/_virtual_includes/kj-async/kj/async-inl.h:1366:3
void
workerd::server::CliMain::serveImpl(workerd::server::CliMain::serve()::'lambda'(workerd::jsg::V8System&,
workerd::server::config::Config::Reader)&&) at src/workerd/server/workerd.c++:1209:15
workerd::server::CliMain::serve() at src/workerd/server/workerd.c++:1221:5
auto workerd::server::CliMain::addServeOptions(kj::MainBuilder&)::'lambda5'(auto&,
auto&&...)::operator()(auto&, auto&&...) at
src/workerd/server/workerd.c++:745:27
auto kj::_::BoundMethod::operator()<>() at
bazel-out/k8-dbg/bin/external/capnp-cpp/src/kj/_virtual_includes/kj/kj/function.h:263:12
kj::MainBuilder::Validity auto
workerd::server::cliMethod>(auto&&)::'lambda'(auto&&...)::operator()<>(auto&&...) at
src/workerd/server/workerd.c++:98:7
```
Contributor guide
Assessment
This issue has not been assessed yet.