michaelfeil / michaelfeil/infinity
Can we ddd a timeout to the requests in the request queue?
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### Feature request
As an embedding service, in scenarios with high QPS (queries per second) and sensitivity to latency, if there are multiple requests piled up in Infinity's request queue and the caller has already timed out, can Infinity discard those requests that have already timed out? This would avoid unnecessary inference. In other words, provide a configuration parameter to set the timeout duration for inference requests in the request queue. If a request times out, it should not be inferred.
### Motivation
"Eliminate invalid requests to improve GPU usage efficiency.
### Your contribution
If adopted,we can work together.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue does not identify files, tests, or an entry point; start by locating the request queue and inference dispatch path. Done means a configurable queue timeout causes expired requests to be discarded before inference, with coverage for the timeout behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100