huggingface / huggingface/inference-benchmarker

[Feature Request] Allow use of non OAI-like API url

Open
#9 1 comment 0 reactions 1 assignee Claimed by @Hugoch View on GitHub
Dominant language
Rust
Stars
172
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Currently the benchmarker can only work with an endpoint accessible through an OAI-like API url set in `--url`. It would be great to extend this to any type of endpoint (e.g [Sagemaker endpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html)) and allow to set up a LLM client that can receive requests from the load tester.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.