cloudflare / cloudflare/workers-sdk
🐛 BUG: Error: Network connection lost.
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.5k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 187
Description
### Which Cloudflare product(s) does this pertain to?
Miniflare
### What versions are you using?
3.20241106.0 [Miniflare], v18.20.4 [Node.js]
### What operating system and version are you using?
Ubuntu 20.04.6 LTS
### Please provide a link to a minimal reproduction
_No response_
### Describe the Bug
Error occurs when a miniflare worker makes a `fetch` request to a worker that is bound via `serviceBindings`.
---
**Problem I am trying to solve:**
I want to make a https request using the built-in workersd `fetch` function, like so:
```
const response = await fetch(`https://abc.xyz.workers.dev`);
```
but it fails with this error:
```
workerd/jsg/util.c++:274: error: e = kj/compat/tls.c++:221: failed: TLS peer's certificate is not trusted; reason = Hostname mismatch
```
Therefore, following the miniflare documentation, I set up a `serviceBinding` to create a worker that invokes a Node.js function. This works well, but at a certain scale it triggers a miniflare or workersd error: `Network connection lost.`
**Reproduction:**
**main.ts**
```
import {Miniflare, Request, Response, Headers} from 'miniflare';
async function EXTERNAL_FETCHER(request: Request): Promise {
try {
const res = await fetch(request.url);
const respHeaders = new Headers(res.headers);
respHeaders.delete('Host');
respHeaders.delete('Content-Length');
return new Response(res.body, {status: res.status, headers: respHeaders});
} catch (e) {
return new Response(`EXTERNAL_FETCHER error: ${(e as Error).toString()}`, {status: 500});
}
}
const mf = new Miniflare({
scriptPath: 'worker.mjs',
https: true,
modules: true,
serviceBindings: {
EXTERNAL_FETCHER,
},
});
try {
const res = await mf.dispatchFetch('http://localhost:8787');
console.log(await res.text()); // Expected: Hello Miniflare!
} catch (e) {
console.error(e);
}
await mf.dispose();
```
**worker.mjs**
```
export default {
async fetch(_request, env) {
const buffers = [];
for (let i = 0; i < 100; i++) {
const response = await env.EXTERNAL_FETCHER.fetch(new Request(
`https://abc.xyz.workers.dev?r2Key=${i}`
));
const buffer = await response.arrayBuffer();
buffers.push(buffer);
}
const uint8Arrays = [];
for (const buffer of buffers) {
const arr = new Uint8Array(buffer);
uint8Arrays.push(arr);
}
for (let i = 0; i < 100; i++) {
const response = await env.EXTERNAL_FETCHER.fetch(new Request(
`https://abc.xyz.workers.dev?r2Key=${i}`
)); // first call fails with Error: Network connection lost.
}
return new Response('Hello Miniflare!');
},
};
```
**Explanation:**
- The cloudflare worker URL`https://abc.xyz.workers.dev` is fake, but when I use my real worker URL it returns binary content of approximately 300kb for each fetch request.
- The `Network connection lost.` error is raised during the the first `fetch` call in the second `for` loop. This does not appear to be related to the number of `fetch` calls that are made, but rather, the size of `uint8Arrays`. By simply `break`ing out of the loop that pushes items onto `uint8Arrays` after 23 iterations, the code works without issues. However, if it reaches 24 iterations, the next subsequent fetch call will fail with the reported error.
**Exploration:**
- This does not appear to be related to Node.js memory limits. After raising the node memory limit to 16gb with the `max-old-space-size` flag, it continues to fail at the same point.
- It works as expected when run outside of miniflare/workersd, with node, deno, and bun, by replacing `await env.EXTERNAL_FETCHER.fetch` with a `fetch` call, as these other runtimes support https.
- It also works inside of miniflare/workersd by changing the url from `http` to `https`, but this is not ideal for my use case, which requires providing an auth token in the URL.
**Hypothesis:**
- My best guess is that this issue is related to either a workersd memory or a bound worker connection time limit. It seems like, at some point the connection to the bound worker is dropped, and after a predefined timeout, the `Network connection lost.` error is returned.
### Please provide any relevant error logs
_No response_
Contributor guide
Research direction
Start with the supplied main.ts and worker.mjs reproduction, running it with the stated Miniflare, Node.js, and Ubuntu versions. Compare behavior when 23 versus 24 binary responses are retained and trace where the subsequent service-binding fetch reports "Network connection lost." Done means the reproduction no longer loses the connection under the described workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100