`http` Manipulate stream

NOTE: I, @josephjclark, am taking over this issue to represent the HTTP streaming problem.

The new HTTP helpers should be able to handle a stream of data and pass them on for further processing.

I don't want to have a special steaming API and special streaming operations. I would prefer the platform to natively understand what a stream is and how to deal with it.

There are three parts to this.

1. Runtime support

The first thing to note here is that we'll need a bunch of support across the runtime AND adaptors to better support streams. There is no point fetching an JSON stream if you have to manually extract the data and dump it all on state. I'm raising an isue for the support stuff HERE.

2. Helper function support

Secondly, the new HTTP helper functions need to be able to return a stream (rather than parsing the data as json or text).

I think we need two support two flags: asStream, which will return the response data as a stream. I'd also like to support the responseType flag - so that if you ask for a JSON stream, we'll pipe the the stream to JSON for you. We could also support csv, tar, tgz and so on.

Now, I may be wrong about the auto-parsing bit. Maybe the helper just returns reponse.body, and the caller then has to do response.body.pipe(jsonParse) or whatever. Because a) why should the function bundle so many functions when all you have to do is response.pipe(), and b) even JSON streaming isn't trivial.

We can stream simple JSON responses - if the response body is an array or an object with key-values, we can stream the array items or the object values. No problem. But if the data has a wrapper, even like { response: 'ok', data: []}, then it's basically unstreamable. So to parse JSON we'd need to take json-array and json-values content types.

So while I'd like to be helpful here, I think I've talked myself into returning the stream as a stream, and leaving the caller to work out how to extract it (this becomes part of #1). Which means stream is a responseType.

One thing the helper DOES need to do is convert the unidici ReadableStream into a node.js Readable (so that we get a familiar pipe function.

3. http adaptor support

Once the underlying helpers are stream friendly, the http adaptor itself needs to reflect streaming in the API.

I think this is just a case of forwarding the responseType: 'stream' from the operation parameters down to the helper function. I don't think we should do auto-parsing or special handling: we just write the stream itself to state.data and let the user manipulate it.

Original issue from @lakhassane

Create a new helper function allowing to:

1. download a stream from a URL,
2. Do treatment on it
3. Upload to a destination system.

Without storing on memory.

Start by locating the new HTTP helper functions and the HTTP adaptor entry point, then trace how operation parameters become state.data. Review the runtime and adaptor stream support around the unidici ReadableStream and Node.js Readable conversion. Done means a response can be returned as a stream, passed through the adaptor, and processed without storing the full payload in memory.