huggingface / huggingface/optimum-benchmark
Possibility to add multiple users / concurrent user requests?
Open
enhancement
- Dominant language
- Python
- Stars
- 341
- Forks
- 64
- Avg merge
- 58m
- Merged PRs (30d)
- 1
Description
Hi there :-)
Is there a possibility to configure multiple users / concurrent request sessions?
I'd like to simulate how the different backends behave if not 1 user, but e.g. 8 users concurrently access the LLM.
I know there is the possibility to configure batches, but there should be a performance difference if e.g. 1 user sends a batch with 8 requests or 8 users independently send a batch with 1 request each. Please correct me if that is not true :-)
Thanks a lot and appreciate the work on optimum-benchmark!
Contributor guide
Assessment
This issue has not been assessed yet.