ValueError: Only able to place 0 replicas, but 2 replicas were requested.

Describe the bug

i want to use deepspeed-fastgen for mixtral-instruct 8*7b inference on multi-node,my deployments are as follows:

import mii
    client = mii.serve("path of mixtral-instruct 8*7b",
            tensor_parallel=16,
            replica_num=2,
            hostfile="tmp/hostfile"
            )
    response = client.generate(["Deepspeed is", "Seattle is"], max_new_tokens=128)
    print(response)

my hostfile like this:

ip1 slots=8
ip2 slots=8

my bash code like this:

deepspeed --hostfile ${TMP_DIR}/hostfile --master_addr ${MASTER_ADDR} --master_port=${MASTER_PORT}  ${WORKSPACE}/eval/eval1.py \
    --deepspeed --deepspeed_config configs/ds_config_zero3.json

the config of bash is working for training

I encountered the following error

• ValueError: Only able to place 0 replicas, but 2 replicas were requested.

how can i fix this errors?

The payload names no repository files or tests. Start by reproducing the mii.serve call with the shown hostfile and tensor-parallel and replica settings, then trace the replica-placement error and compare the requested resources with the available slots. Done means two replicas start successfully and the generation call returns results across the multi-node deployment.