docs: document latency predictor sidecar and LoRA affinity scorer auto-injection
- Dominant language
- MDX
- Stars
- 113
- Forks
- 193
- Avg merge
- 5d 20h
- Merged PRs (30d)
- 3
Description
## Summary
KServe v0.20 added two auto-injection features for LLMInferenceService that are not documented:
1. **Latency predictor sidecar**: When the `predicted-latency-producer` plugin is detected in the EPP config, the controller auto-injects training-server and prediction-server sidecar containers via a well-known `LLMInferenceServiceConfig`.
2. **LoRA affinity scorer**: When LoRA adapters are configured, the `lora-affinity-scorer` plugin is automatically injected into the default scheduling profile, scoring endpoints by whether the requested adapter is already active on each pod.
## What needs to be documented
### Latency predictor sidecar
- How to enable: add `predicted-latency-producer` plugin to EPP inline config
- Auto-injection of `kserve-config-llm-scheduler-latency-predictor` well-known config
- What gets injected: training-server and prediction-server sidecar containers
- How to override defaults via custom `LLMInferenceServiceConfig` and `baseRefs`
- Example minimal scheduler config
### LoRA affinity scorer
- Automatic injection when `spec.model.lora.adapters` is configured
- What it does: scores endpoints by active adapter availability
- Injection details: added to `plugins` list, `pluginRef` with `weight: 4` in default scheduling profile
- No manual configuration required — it's fully automatic
- Update existing LoRA adapter documentation to mention this behavior
## Related PRs
- https://github.com/kserve/kserve/pull/5678 (latency predictor sidecar)
- https://github.com/kserve/kserve/pull/5655 (LoRA affinity scorer)
## Since
KServe v0.20
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing scheduler and LoRA adapter documentation, then review related PRs #5678 and #5655 for the latency predictor sidecar and LoRA affinity scorer behavior. Document the listed enablement, injection, override, and scheduling details, including a minimal scheduler configuration example. Done means both features and the existing LoRA documentation clearly explain their automatic behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100