Wrong shm size in deployment pod when inference model with multi gpu

I caught an error when deploy with multi gpu inference

torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5 ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. Last error: Error while creating shared memory segment /dev/shm/nccl-F9ffgB (size 9637888)

I think the error came from the shm size. Then i have defined shmSize in yaml config below:
`servingEngineSpec:
runtimeClassName: ""
modelSpec:

• name: "opt125m"
  repository: "vllm/vllm-openai"
  tag: "latest"
  modelURL: "facebook/opt-125m"

  replicaCount: 1

  requestCPU: 6
  requestMemory: "16Gi"
  requestGPU: 2

  vllmConfig:
  maxModelLen: 200
  dtype: "bfloat16"
  extraArgs: ["--disable-log-requests", "--gpu-memory-utilization", "0.1", "--tensor-parallel-size", "2"]

  shmSize: "20Gi"`

Then i ran helm install again with this config, but i still caught the same error. When i check shm size in deployment pod, it showed that my shm size in container pod only has 64MB:
kubectl exec -it vllm-opt125m-deployment-vllm-8548d65cf6-blt5m -- df -h /dev/shm Filesystem Size Used Avail Use% Mounted on shm 64M 0 64M 0% /dev/shm

How can i deal with that? Thanks in advance
cc: @YuhanLiu11