huggingface / huggingface/inference-benchmarker
[Feature Request] Allow use of non OAI-like API url
Open
- Dominant language
- Rust
- Stars
- 172
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Currently the benchmarker can only work with an endpoint accessible through an OAI-like API url set in `--url`. It would be great to extend this to any type of endpoint (e.g [Sagemaker endpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html)) and allow to set up a LLM client that can receive requests from the load tester.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.