high latency

Latency in OpenVINO Model Server Inside Kubernetes Cluster

To Reproduce

Steps to reproduce the behavior:

1. Prepare Models Repository: Followed standard procedures to set up the models repository.
2. OVMS Launch Command:

   docker run -d --rm \
     -v /home/raa/model:/models \
     -p 900:9000 \
     -p 800:8000 \
     openvino/model_server:latest \
     --model_path /models/model1/1 \
     --model_name model1 \
     --port 9000 \
     --rest_port 8000 \
     --log_level DEBUG
   

3. **Client command ** :

    server address = 192.168.1.10:900
    client = make_grpc_client(server_address)
    pred = client.predict(inputs=inputs, model_name=model1)
   

Environment

I am using OpenVINO server for model deployment on edge devices with an Intel i7-13th Gen processor. My PyTorch model is trained with an image size of 640.

• Standalone Inference: The prediction time for PyTorch weights is consistently <0.01 seconds per image.
• IR Converted Weights: Similar inference time in standalone mode.
• In Kubernetes Cluster: When deployed, the OpenVINO server takes approximately 1.5 seconds per prediction using gRPC calls.
• Observation: Running the OpenVINO server outside the cluster in a container yields <0.01 seconds inference time for IR converted weights, while the same server inside the cluster significantly increases latency.

Expected Behavior

I expected the inference time for the OpenVINO Model Server inside the Kubernetes cluster to be comparable to that observed outside the cluster (i.e., <0.01 seconds).

model directory format

model/
├── model1/
│   └── 1/
│       ├── model.bin
│       ├── model.mapping
│       └── model.xml

Start with the provided Docker launch command and gRPC client, then compare their timing with the Kubernetes deployment and its networking or runtime configuration. Reproduce the gap between sub-0.01-second standalone inference and 1.5-second cluster predictions; done means the cause is identified and comparable latency is restored or the limitation is documented.