RetryAgent - Always Throws Error on Status Codes

This would solve...

Avoid losing the response body when a status code defined in the RetryAgent is returned after retry attempts are exhausted. Instead of throwing a RequestRetryError immediately after checking if the statusCode is in the list, the agent could just let the request finish, allowing the response body to be read.

The implementation should be like this...

Introduce a new option, for example: noThrowStatusCodes, which accepts an array of status codes.

If the response status code matches any code in this array, the agent would not throw a RequestRetryError and would continue, allowing the response body to be processed.

const retryAgent = new RetryAgent({
    // Continue the request without throwing an error if the cause is a 503 status code.
    // But throw an error if it's a 500 status code and retry attempts are finished
    noThrowStatusCodes: [503],
    statusCodes: [503, 500],
})

I also considered...

Adding the body directly to the RequestRetryError, but it didn't seem like a good idea.

Context

My use case involves scraping data from web applications standardized by a website builder. There are thousands of these applications with the same format.

In some cases, this application returns a 500+ status code, even though the response body contains valid data that I want to capture. This inconsistency comes from the website builder model, but it's something I need to deal with.

I had been using the RetryAgent for a while, but recently I started facing issues when the website builder began returning 500+ status codes on some requests, causing real impacts on my workflow.

Showcase

In the showcase, I have more details to explain the problem. Feel free to access the repository and check the code.

https://github.com/sydo26/retry-agent-always-throws-error-showcase

Final considerations

If there is any other alternative to solve the problem besides the proposed one, please let me know. I am open to suggestions.

I saw this issue opened (November 2024) https://github.com/nodejs/undici/issues/3837, and it seemed similar to the problem I am facing but with a different solution that does not apply to my case.

Start by reading the RetryAgent implementation and the handling of configured statusCodes after retry attempts are exhausted. Check the existing RetryAgent tests, if present, to understand current error behavior and add coverage for the proposed opt-out status-code behavior. Done means the response body remains readable for opted-out codes while other configured codes still raise the existing error.