nodejs / nodejs/undici

Undici's Fetch's Body takes a long time to load compared to just request

Open
#2,014 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
JavaScript
Stars
7.7k
Forks
880
Avg merge
2d 16h
Merged PRs (30d)
68

Description

Bug Description

I've noticed a bug where if the body of a fetch response is large (let us say above 50mb), grabbing the body contents takes a long time to load. I compared it to undici's normal request function and node-fetch's fetch response. In comparison, node-fetch took about 8.5 to 9.4 seconds to grab a 130mb file, undici's request function took 2.3 to 2.6 seconds, and undici's fetch took 95.5 to 104.2 seconds.

Reproducible By

undici fetch, 95.5 seconds, 18,605 buffers

await (async () => {
    const { fetch } = require('undici');

    const response = await fetch('https://alpha.notsobot.com/api/undici/test');
    let now = Date.now();
    const buffers = [];
    for await (let x of response.body) {
        buffers.push(x);
    }
    console.log(Date.now() - now);
    now = Date.now();
    JSON.parse(Buffer.concat(buffers));
    console.log(Date.now() - now);
})();

undici request, 2.6 seconds, 9794 buffers

await (async () => {
    const { request } = require('undici');

    const response = await request('https://alpha.notsobot.com/api/undici/test');
    let now = Date.now();
    const buffers = [];
    for await (let x of response.body) {
        buffers.push(x);
    }
    console.log(Date.now() - now);
    now = Date.now();
    JSON.parse(Buffer.concat(buffers));
    console.log(Date.now() - now);
})();

node-fetch fetch, 9.4 seconds, 28909 buffers

await (async () => {
    const fetch = require('node-fetch');
    const response = await fetch('https://alpha.notsobot.com/api/undici/test');
    let now = Date.now();
    const buffers = [];
    for await (let x of response.body) {
        buffers.push(x);
    }
    console.log(Date.now() - now);
    now = Date.now();
    JSON.parse(Buffer.concat(buffers));
    console.log(Date.now() - now);
})();

(Also, doing .arrayBuffer() yields the same results)

Expected Behavior

Similar timing to undici's request function

Logs & Screenshots

Environment

image

Ubuntu 22.10, Node v19.8.1 and v18.7.0, undici v5.2.0 and v5.21.0

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided large-response comparison for undici fetch, undici request, and node-fetch, including the response.body and arrayBuffer cases. Trace the fetch body-consumption path against request's response.body handling; done means large bodies load with timing comparable to the expected request behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.