drogonframework / drogonframework/drogon
Overhead of creating new threads in Streaming Response
- Dominant language
- C++
- Stars
- 14.3k
- Forks
- 1.4k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 14
Description
**Is your feature request related to a problem? Please describe.**
There is an overhead of creating new threads when using streaming response feature.
This drogon example demonstrates it very well: https://github.com/drogonframework/drogon/blob/8541e671430f9605a62f1afbd786c1dbd54197d6/examples/async_stream/main.cc#L15-L24
We have uncontrolled number of new threads being created which is a bottleneck for high-throughput workloads.
Another issue is that the thread is detached and never joined.
**Describe the solution you'd like**
Other http frameworks use internal event loop to schedule partial send operation, example:
https://github.com/tensorflow/serving/blob/8eec641dee627243e13f8d04d8980f326c3e371d/tensorflow_serving/util/net_http/server/internal/evhttp_request.cc#L359-L360
**Describe alternatives you've considered**
No alternative
**Additional context**
High-throughput scenarios, for example serving LLM text generation with streaming output approach
Contributor guide
Assessment
This issue has not been assessed yet.