Add an option to limit the max number of desired session counts for inference endpoints
Open
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
If the user increases it to a very large number, it may exceeds the system limit to handle the pending session queue.
This would be also further constrained by #1120 as the actual limit should be the minimum of both settings: (the pending session limit, the desired session count limit).
JIRA Issue: BA-155
Contributor guide
Assessment
This issue has not been assessed yet.