Significant Slowdown in Requests When Using ProxyAgent
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 7.7k
- Forks
- 880
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 68
Description
Hello,
I'm experiencing a significant slowdown in request performance when using ProxyAgent with the undici library. The same requests without a proxy are much faster. Below are the details of my setup and the observed behavior.
Steps to Reproduce:
- Set up a basic HTTP client using undici:
import {
Client,
Dispatcher,
request,
ProxyAgent,
getGlobalDispatcher,
} from "undici";
import { Readable, Writable } from "stream";
import { IncomingHttpHeaders } from "undici/types/header";
import { createGunzip } from "zlib";
import { promisify } from "util";
import { Buffer } from "buffer";
const pipeline = promisify(require("stream").pipeline);
export class HttpClient {
private baseURL: string;
private defaultHeaders: Record<string, string> = {};
private proxy: string | undefined;
private timeout: number = 30000;
private dispatcher: Dispatcher;
constructor(baseURL: string, proxyneeded: boolean = true, timeout?: number) {
this.baseURL = baseURL;
if (timeout) {
this.timeout = timeout;
}
if (proxyneeded) {
this.proxy = "http://127.0.0.1:3128";
this.dispatcher = new ProxyAgent(this.proxy);
} else {
this.dispatcher = getGlobalDispatcher();
}
}
public get<T = any>(
url: string,
config?: RequestInit & { params?: Record<string, any> }
): Promise<T> {
return this.request<T>({
...config,
method: "GET",
path: this.buildUrl(url, config?.params),
});
}
public post<T = any>(
url: string,
data?: any,
config?: RequestInit
): Promise<T> {
return this.request<T>({
...config,
method: "POST",
path: url,
body: data,
});
}
public put<T = any>(
url: string,
data?: any,
config?: RequestInit
): Promise<T> {
return this.request<T>({ ...config, method: "PUT", path: url, body: data });
}
public delete<T = any>(
url: string,
config?: RequestInit & { params?: Record<string, any> }
): Promise<T> {
return this.request<T>({
...config,
method: "DELETE",
path: this.buildUrl(url, config?.params),
});
}
public request<T = any>(
config: RequestInit & { path: string }
): Promise<T> {
const { method, path, body, headers } = config;
if (!method) {
return Promise.reject(new Error("HTTP method is required"));
}
const url = !path.startsWith("/") ? `/${path}` : path;
return request(this.baseURL + url, {
method: method as Dispatcher.HttpMethod,
body: body ? JSON.stringify(body) : undefined,
headers: {
...this.defaultHeaders,
...headers,
} as
| IncomingHttpHeaders
| string[]
| Iterable<[string, string | string[] | undefined]>
| null,
headersTimeout: this.timeout,
dispatcher: this.dispatcher,
} as any)
.then(({ statusCode, headers, trailers, body }: any) => {
return this.getResponseBody(headers, body);
})
.then(responseContent => {
return { data: responseContent } as T;
});
}
public setHeader(name: string, value: string): void {
this.defaultHeaders[name] = value;
}
public removeHeader(name: string): void {
delete this.defaultHeaders[name];
}
private buildUrl(url: string, params?: Record<string, any>): string {
if (!params) {
return url;
}
const urlObj = new URL(url, this.baseURL);
Object.keys(params).forEach((key) =>
urlObj.searchParams.append(key, params[key])
);
return urlObj.toString();
}
private getResponseBody(
headers: IncomingHttpHeaders,
response: Dispatcher.BodyMixin
): Promise<any> {
const buffers: Uint8Array[] = [];
return new Promise((resolve, reject) => {
(async () => {
try {
for await (const chunk of response as any) {
buffers.push(chunk);
}
const buffer = Buffer.concat(buffers);
if (headers["content-encoding"] === "gzip") {
this.decompressGzip(buffer).then(resolve).catch(reject);
} else {
resolve(this.parseResponse(buffer));
}
} catch (err) {
reject(err);
}
})();
});
}
private decompressGzip(buffer: Buffer): Promise<any> {
const gunzip = createGunzip();
const decompressedBuffers: Buffer[] = [];
return new Promise((resolve, reject) => {
pipeline(
Readable.from([buffer]),
gunzip,
new Writable({
write(chunk, encoding, callback) {
decompressedBuffers.push(chunk);
callback();
},
})
)
.then(() => {
const decompressed = Buffer.concat(decompressedBuffers);
resolve(this.parseResponse(decompressed));
})
.catch(reject);
});
}
private parseResponse(buffer: Buffer): any {
const responseText = buffer.toString();
try {
return JSON.parse(responseText);
} catch {
return responseText;
}
}
}
- Compare the response time when using ProxyAgent versus without using any proxy.
Thank you for your attention to this issue. Any insights or fixes would be greatly appreciated.
Best regards,
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue provides a JavaScript/TypeScript HttpClient reproduction using ProxyAgent and getGlobalDispatcher, but names no repository files or tests. Start by running the supplied comparison with and without the proxy, then trace the ProxyAgent request path. Done means the slowdown is explained and a regression test demonstrates acceptable proxy request performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100