kserve / kserve/website

docs: document latency predictor sidecar and LoRA affinity scorer auto-injection

Open
#740 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
113
Forks
193
Avg merge
5d 20h
Merged PRs (30d)
3

Description

## Summary

KServe v0.20 added two auto-injection features for LLMInferenceService that are not documented:

1. **Latency predictor sidecar**: When the `predicted-latency-producer` plugin is detected in the EPP config, the controller auto-injects training-server and prediction-server sidecar containers via a well-known `LLMInferenceServiceConfig`.

2. **LoRA affinity scorer**: When LoRA adapters are configured, the `lora-affinity-scorer` plugin is automatically injected into the default scheduling profile, scoring endpoints by whether the requested adapter is already active on each pod.

## What needs to be documented

### Latency predictor sidecar
- How to enable: add `predicted-latency-producer` plugin to EPP inline config
- Auto-injection of `kserve-config-llm-scheduler-latency-predictor` well-known config
- What gets injected: training-server and prediction-server sidecar containers
- How to override defaults via custom `LLMInferenceServiceConfig` and `baseRefs`
- Example minimal scheduler config

### LoRA affinity scorer
- Automatic injection when `spec.model.lora.adapters` is configured
- What it does: scores endpoints by active adapter availability
- Injection details: added to `plugins` list, `pluginRef` with `weight: 4` in default scheduling profile
- No manual configuration required — it's fully automatic
- Update existing LoRA adapter documentation to mention this behavior

## Related PRs

- https://github.com/kserve/kserve/pull/5678 (latency predictor sidecar)
- https://github.com/kserve/kserve/pull/5655 (LoRA affinity scorer)

## Since

KServe v0.20

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing scheduler and LoRA adapter documentation, then review related PRs #5678 and #5655 for the latency predictor sidecar and LoRA affinity scorer behavior. Document the listed enablement, injection, override, and scheduling details, including a minimal scheduler configuration example. Done means both features and the existing LoRA documentation clearly explain their automatic behavior.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.