kserve / kserve/website

docs: add multi-node inference without Ray (mp backend) guide

Open
#722 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
113
Forks
193
Avg merge
5d 20h
Merged PRs (30d)
3

Description

## Summary

KServe v0.18 added support for multi-node InferenceService deployments **without requiring Ray**, using vLLM's `mp` (multiprocessing) distributed executor backend. The current multi-node documentation at `docs/model-serving/generative-inference/multi-node/multi-node.md` only covers Ray-based multi-node inference.

## What needs to be documented

- The `multinode/executor-backend` annotation on ServingRuntime spec (`ray` vs `mp` mode)
- How `mp` mode derives the number of nodes from pipeline parallelism (PP) and GPUs per node from tensor parallelism (TP)
- The worker headless service for pod DNS discovery
- Environment variables injected into worker containers (`PIPELINE_PARALLEL_SIZE`, `TENSOR_PARALLEL_SIZE`, `WORKER_SVC`)
- Example YAML for a multi-node InferenceService using `mp` backend

## Related PR

- https://github.com/kserve/kserve/pull/5366

## Since

KServe v0.18

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/model-serving/generative-inference/multi-node/multi-node.md and the related PR #5366 to understand the existing Ray-based guide. Document the mp annotation, PP/TP sizing, worker headless service, injected environment variables, and add a complete multi-node InferenceService YAML example; done means the guide covers KServe v0.18 mp deployments without Ray.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.