Azure / Azure/azure-functions-nodejs-library
HTTP Streams, not possible to implement backpressure or to detect when a client disconnects
- Dominant language
- TypeScript
- Stars
- 70
- Forks
- 35
- Avg merge
- 4d 18h
- Merged PRs (30d)
- 6
Description
I am writing a function that retrieves the files from Azure Blob Storage and returns a zip file containing them. HTTP Streams seem like a perfect match for this use case: I can just start loading files one by one from storage and stream the archive to the client on the fly.
The proof of concept works, but I'm facing the following issues:
1. There seems to be no way to detect that a client closed the connection. The function keeps writing data *somewhere*, and only when everything is written, there appears an error in the log (`[Error] Executed '' (Failed, Id=, Duration=54018ms)`)
2. There is no way to implement backpressure. The "drain" event is never emitted on the stream that I return in the response body, the data is consumed immediately and fully whenever I pass it into the stream. I tested with slow clients: the function keeps sending data, while the memory used by the function app rapidly increases. Out of curiosity I looked around the codebase, it seems the reason is [this code](https://github.com/Azure/azure-functions-nodejs-library/blob/v4.x/src/http/httpProxy.ts#L50-L52), where the response is written unconditionally, without checking the value returned by [ServerResponse.write](https://nodejs.org/api/http.html#responsewritechunk-encoding-callback).
For completeness, this is the PoC code I have:
```typescript
import { InvocationContext, HttpHandler, HttpRequest } from '@azure/functions';
import * as archiver from "archiver";
import * as stream from "stream";
export const Download: HttpHandler = async (request: HttpRequest, context: InvocationContext) => {
const ptStream = new stream.PassThrough({ highWaterMark: 64 * 1024 });
ptStream.on("drain", () => {
context.log("This is never called");
});
const archive = archiver("zip", { zlib: { level: 1 } });
archive.pipe(ptStream);
const queue = [...input.filenames];
const processNext = async () => {
const filename = queue.pop();
const blockBlobClient = containerClient.getBlockBlobClient(filename);
const downloadResponse = await blockBlobClient.download();
const stream = downloadResponse.readableStreamBody as NodeJS.ReadableStream;
archive.append(stream, { name: filename });
};
archive.on('entry', (e) => {
if (queue.length == 0) {
return archive.finalize();
}
processNext();
});
processNext();
return {
body: ptStream,
status: 200,
headers: {
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename=${input.archiveName}`
}
}
}
```
Contributor guide
Assessment
This issue has not been assessed yet.