huggingface / huggingface/inference-benchmarker

Benchmark fails at the end and do not create json file test result

Open
#27 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
172
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Why is some occasion the tool seems to fail at the end of the entire test run and does not generate the json output file?

Here is an example of my launcher command:
```
docker run --rm -it --net host -v $(pwd):/opt/inference-benchmarker/results -e "HF_TOKEN=$HF_TOKEN" ghcr.io/huggingface/inference-benchmarker:latest inference-benchmarker --tokenizer-name "$MODEL" --url http://localhost:8080/ --profile chat
```
I run distributed vllm on a 4-node Ray cluster

```
VLLM_HOST_IP=${HEAD_NODE_IP} python3 -m vllm.entrypoints.openai.api_server \
--model ${MODEL_NAME} \
--host 0.0.0.0 \
--port ${VLLM_PORT} \
--tensor-parallel-size 4 \
--pipeline-parallel-size 8 \
--max-model-len 4096 \
--max-num-batched-tokens 8192 \
--max-num-seqs 256 \
--disable-custom-all-reduce \
--distributed-executor-backend ray"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.