yet another - too big subrequest response while sending to client - issue

I need to transform response body so I am using r.subrequest which runs into too big subrequest response while sending to client. Several issues are there to this effect and the solution seems to be to use

subrequest_output_buffer_size {size};

In my case I kept increasing this value until 4M until it worked by trial and error.

My question is what should be this value - does this depend on the size of the response.
If it is then how can this value to be set when we may not the size upfront.
I understand that setting this value too big is safe for it to work but at the same time unsafe for resources etc.

I tried using responseBuffer instead of responseBody but of no use.

Wondering if I should/should use subrequest in the first place ?

Wondering if this has anything to do with internal; in the subrequest location ?

# nginx.conf
location /search-subrequest {
     proxy_pass https://redacted-domain/api/search;
     subrequest_output_buffer_size 4M; # 4M is big
}
location /search {

  js_content main.search;

and here is my njs

// search.js

async function search(r) {
    let reply = await r.subrequest('/search-subrequest');
    let bodyRaw = reply.responseBuffer.toString('utf8');
    let body = JSON.parse(bodyRaw);
    // transform body
    r.return(200, JSON.stringify(body));
 }

and the errors curl -I http://localhost/search

curl: (52) Empty reply from server
[warn] 415#415: *142 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/02/0000000021 while reading upstream, client: 127.0.0.1, server: localhost, request: "HEAD / HTTP/1.1", subrequest: "/search-subrequest", upstream: "https://REACTTEDIP:443/api/search", host: "localhost"

[error] 415#415: *142 too big subrequest response while sending to client, client: 127.0.0.1, server: localhost, request: "HEAD / HTTP/1.1", subrequest: "/search-subrequest", upstream: "https://REACTTEDIP:443/api/search", host: "localhost"

Start with the nginx.conf subrequest location and search.js, then reproduce the reported error using the r.subrequest call and responseBuffer handling. Read the subrequest_output_buffer_size behavior and related NGINX subrequest guidance; done would require a clear, documented recommendation addressing buffer sizing, resource implications, and whether this usage is appropriate.