nodejs / nodejs/undici

Significant Slowdown in Requests When Using ProxyAgent

Open
#3,403 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
7.7k
Forks
880
Avg merge
2d 16h
Merged PRs (30d)
68

Description

Hello,

I'm experiencing a significant slowdown in request performance when using ProxyAgent with the undici library. The same requests without a proxy are much faster. Below are the details of my setup and the observed behavior.

Steps to Reproduce:
  1. Set up a basic HTTP client using undici:
import {
  Client,
  Dispatcher,
  request,
  ProxyAgent,
  getGlobalDispatcher,
} from "undici";
import { Readable, Writable } from "stream";
import { IncomingHttpHeaders } from "undici/types/header";
import { createGunzip } from "zlib";
import { promisify } from "util";
import { Buffer } from "buffer";

const pipeline = promisify(require("stream").pipeline);

export class HttpClient {
  private baseURL: string;
  private defaultHeaders: Record<string, string> = {};
  private proxy: string | undefined;
  private timeout: number = 30000;
  private dispatcher: Dispatcher;

  constructor(baseURL: string, proxyneeded: boolean = true, timeout?: number) {
    this.baseURL = baseURL;

    if (timeout) {
      this.timeout = timeout;
    }

    if (proxyneeded) {
      this.proxy = "http://127.0.0.1:3128";
      this.dispatcher = new ProxyAgent(this.proxy);
    } else {
      this.dispatcher = getGlobalDispatcher();
    }
  }

  public get<T = any>(
    url: string,
    config?: RequestInit & { params?: Record<string, any> }
  ): Promise<T> {
    return this.request<T>({
      ...config,
      method: "GET",
      path: this.buildUrl(url, config?.params),
    });
  }

  public post<T = any>(
    url: string,
    data?: any,
    config?: RequestInit
  ): Promise<T> {
    return this.request<T>({
      ...config,
      method: "POST",
      path: url,
      body: data,
    });
  }

  public put<T = any>(
    url: string,
    data?: any,
    config?: RequestInit
  ): Promise<T> {
    return this.request<T>({ ...config, method: "PUT", path: url, body: data });
  }

  public delete<T = any>(
    url: string,
    config?: RequestInit & { params?: Record<string, any> }
  ): Promise<T> {
    return this.request<T>({
      ...config,
      method: "DELETE",
      path: this.buildUrl(url, config?.params),
    });
  }

  public request<T = any>(
    config: RequestInit & { path: string }
  ): Promise<T> {
    const { method, path, body, headers } = config;

    if (!method) {
      return Promise.reject(new Error("HTTP method is required"));
    }

    const url = !path.startsWith("/") ? `/${path}` : path;

    return request(this.baseURL + url, {
      method: method as Dispatcher.HttpMethod,
      body: body ? JSON.stringify(body) : undefined,
      headers: {
        ...this.defaultHeaders,
        ...headers,
      } as
        | IncomingHttpHeaders
        | string[]
        | Iterable<[string, string | string[] | undefined]>
        | null,
      headersTimeout: this.timeout,
      dispatcher: this.dispatcher,
    } as any)
      .then(({ statusCode, headers, trailers, body }: any) => {
        return this.getResponseBody(headers, body);
      })
      .then(responseContent => {
        return { data: responseContent } as T;
      });
  }

  public setHeader(name: string, value: string): void {
    this.defaultHeaders[name] = value;
  }

  public removeHeader(name: string): void {
    delete this.defaultHeaders[name];
  }

  private buildUrl(url: string, params?: Record<string, any>): string {
    if (!params) {
      return url;
    }

    const urlObj = new URL(url, this.baseURL);
    Object.keys(params).forEach((key) =>
      urlObj.searchParams.append(key, params[key])
    );
    return urlObj.toString();
  }

  private getResponseBody(
    headers: IncomingHttpHeaders,
    response: Dispatcher.BodyMixin
  ): Promise<any> {
    const buffers: Uint8Array[] = [];
    return new Promise((resolve, reject) => {
      (async () => {
        try {
          for await (const chunk of response as any) {
            buffers.push(chunk);
          }
          const buffer = Buffer.concat(buffers);

          if (headers["content-encoding"] === "gzip") {
            this.decompressGzip(buffer).then(resolve).catch(reject);
          } else {
            resolve(this.parseResponse(buffer));
          }
        } catch (err) {
          reject(err);
        }
      })();
    });
  }

  private decompressGzip(buffer: Buffer): Promise<any> {
    const gunzip = createGunzip();
    const decompressedBuffers: Buffer[] = [];
    return new Promise((resolve, reject) => {
      pipeline(
        Readable.from([buffer]),
        gunzip,
        new Writable({
          write(chunk, encoding, callback) {
            decompressedBuffers.push(chunk);
            callback();
          },
        })
      )
        .then(() => {
          const decompressed = Buffer.concat(decompressedBuffers);
          resolve(this.parseResponse(decompressed));
        })
        .catch(reject);
    });
  }

  private parseResponse(buffer: Buffer): any {
    const responseText = buffer.toString();
    try {
      return JSON.parse(responseText);
    } catch {
      return responseText;
    }
  }
}
  1. Compare the response time when using ProxyAgent versus without using any proxy.

Thank you for your attention to this issue. Any insights or fixes would be greatly appreciated.

Best regards,

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue provides a JavaScript/TypeScript HttpClient reproduction using ProxyAgent and getGlobalDispatcher, but names no repository files or tests. Start by running the supplied comparison with and without the proxy, then trace the ProxyAgent request path. Done means the slowdown is explained and a regression test demonstrates acceptable proxy request performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.