kserve / kserve/website

docs: document model name-based routing for LLMInferenceService

Open
#730 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
113
Forks
193
Avg merge
5d 20h
Merged PRs (30d)
3

Description

## Summary

KServe v0.19 added model name-based routing for LLMInferenceService, enabling shared-gateway deployments. While the Envoy AI Gateway integration documents `x-ai-eg-model` header routing, the generic `X-Gateway-Model-Name` header routing is not documented.

## What needs to be documented

- How model name-based routing works with the `X-Gateway-Model-Name` header
- Configurable header name
- How LoRA adapter requests are automatically routed through the same backend by dynamically expanding header matches
- Shared-gateway deployment pattern where a single gateway dispatches to multiple models
- Integration with components like llm-d BBR (IPP) or gateway-specific native features
- `-model-routing` suffix address types in status

## Related PR

- https://github.com/kserve/kserve/pull/5521

## Since

KServe v0.19

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the related PR #5521 and the existing Envoy AI Gateway integration documentation. Document generic X-Gateway-Model-Name routing, configurable header names, LoRA adapter routing, shared-gateway deployments, llm-d BBR or gateway-native integrations, and -model-routing status addresses; done means each listed behavior is covered.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.