vllm-project / vllm-project/aibrix

Add probe usage practice for super large models, including multi-node case

Open
#782 0 comments 0 reactions 1 assignee Claimed by @Jeffwan View on GitHub
area/performance kind/documentation kind/enhancement priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

When we deploy deepseek 671B model using multi-node way, start up takes very long. It brings few problems
1. It's better to use `startupProbe` and `livenessProbe`, `readinessProbe` to control the interval separately.
2. ray cluster probe can be managed and injected by ray cluster controller. this is helpful for ray cluster controller to manage raycluster in fault tolerant way. however, we care more about the application status vLLM.

We need to build some practice on this, how to make two mechanisms work together or just use application one instead.

### Use Case

fault tolerance and high availability

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.