huggingface / huggingface/optimum-benchmark

Possibility to add multiple users / concurrent user requests?

Open
#222 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
341
Forks
64
Avg merge
58m
Merged PRs (30d)
1

Description

Hi there :-)

Is there a possibility to configure multiple users / concurrent request sessions?
I'd like to simulate how the different backends behave if not 1 user, but e.g. 8 users concurrently access the LLM.

I know there is the possibility to configure batches, but there should be a performance difference if e.g. 1 user sends a batch with 8 requests or 8 users independently send a batch with 1 request each. Please correct me if that is not true :-)

Thanks a lot and appreciate the work on optimum-benchmark!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.