michaelfeil / michaelfeil/infinity

Can we ddd a timeout to the requests in the request queue?

Open
#546 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### Feature request

As an embedding service, in scenarios with high QPS (queries per second) and sensitivity to latency, if there are multiple requests piled up in Infinity's request queue and the caller has already timed out, can Infinity discard those requests that have already timed out? This would avoid unnecessary inference. In other words, provide a configuration parameter to set the timeout duration for inference requests in the request queue. If a request times out, it should not be inferred.

### Motivation

"Eliminate invalid requests to improve GPU usage efficiency.

### Your contribution

If adopted,we can work together.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not identify files, tests, or an entry point; start by locating the request queue and inference dispatch path. Done means a configurable queue timeout causes expired requests to be discarded before inference, with coverage for the timeout behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.