docs: add multi-node inference without Ray (mp backend) guide
- Dominant language
- MDX
- Stars
- 113
- Forks
- 193
- Avg merge
- 5d 20h
- Merged PRs (30d)
- 3
Description
## Summary
KServe v0.18 added support for multi-node InferenceService deployments **without requiring Ray**, using vLLM's `mp` (multiprocessing) distributed executor backend. The current multi-node documentation at `docs/model-serving/generative-inference/multi-node/multi-node.md` only covers Ray-based multi-node inference.
## What needs to be documented
- The `multinode/executor-backend` annotation on ServingRuntime spec (`ray` vs `mp` mode)
- How `mp` mode derives the number of nodes from pipeline parallelism (PP) and GPUs per node from tensor parallelism (TP)
- The worker headless service for pod DNS discovery
- Environment variables injected into worker containers (`PIPELINE_PARALLEL_SIZE`, `TENSOR_PARALLEL_SIZE`, `WORKER_SVC`)
- Example YAML for a multi-node InferenceService using `mp` backend
## Related PR
- https://github.com/kserve/kserve/pull/5366
## Since
KServe v0.18
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with docs/model-serving/generative-inference/multi-node/multi-node.md and the related PR #5366 to understand the existing Ray-based guide. Document the mp annotation, PP/TP sizing, worker headless service, injected environment variables, and add a complete multi-node InferenceService YAML example; done means the guide covers KServe v0.18 mp deployments without Ray.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100