drogonframework / drogonframework/drogon

Overhead of creating new threads in Streaming Response

Open
#2,201 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
14.3k
Forks
1.4k
Avg merge
1d 13h
Merged PRs (30d)
14

Description

**Is your feature request related to a problem? Please describe.**
There is an overhead of creating new threads when using streaming response feature.
This drogon example demonstrates it very well: https://github.com/drogonframework/drogon/blob/8541e671430f9605a62f1afbd786c1dbd54197d6/examples/async_stream/main.cc#L15-L24

We have uncontrolled number of new threads being created which is a bottleneck for high-throughput workloads.
Another issue is that the thread is detached and never joined.

**Describe the solution you'd like**
Other http frameworks use internal event loop to schedule partial send operation, example:
https://github.com/tensorflow/serving/blob/8eec641dee627243e13f8d04d8980f326c3e371d/tensorflow_serving/util/net_http/server/internal/evhttp_request.cc#L359-L360

**Describe alternatives you've considered**
No alternative

**Additional context**
High-throughput scenarios, for example serving LLM text generation with streaming output approach

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.