Kong / Kong/developer.konghq.com

feat(ai-gateway): Semantic routing: document index lifecycle, stale-index failure modes, dimensions alignment, and description guidance

Open
#5,384 0 comments 0 reactions 0 assignees View on GitHub
internal product:ai-gateway
Dominant language
Ruby
Stars
28
Forks
121
Avg merge
1d 4h
Merged PRs (30d)
313

Description

## Overview

We need to address the following documentation gaps:

**1. Index lifecycle — missing across all docs.**
Neither the semantic-similarity reference nor the `ai-proxy-advanced` plugin page explains how the vector index is built or maintained for semantic routing. Needs to be documented somewhere (likely both, or a dedicated troubleshooting/reference section): the plugin embeds each target's `description` at initialization and stores vectors keyed by a hash of the target config. If a `description` changes and the plugin considers the existing index valid (logs `[redis] existing index matches config`), it does not re-embed. The stale vector still exists under the old hash, which no longer maps to any live target. At request time this produces `balance.lua:166: found target with ID … but it's not mapped to a target` and a `503 "No peers are available"`. An operator who just edited a description and redeployed has no indication anything is wrong until requests start failing.

**2. Catch-all behavior on empty or stale index — missing across all docs.**
Related to the above but distinct: if the index exists but contains no valid vectors (zero docs, or only stale ones), prompts fall silently to the catch-all with a normal `200` response. There is no error, no log at a visible level, nothing client-side. The semantic-similarity page mentions the catch-all only indirectly. This failure mode needs an explicit callout: what it looks like, how to detect it (check `num_docs` in Redis vs. the number of configured targets), and what state produces it.

**3. `dimensions` must match the embedding model's native output — gap in existing content.**
The semantic-similarity page has a solid general section on dimensionality tradeoffs, but it does not say that the `dimensions` value in the plugin config must equal what the embedding model actually outputs. If `dimensions` is set lower than the model's native size (for example, `768` against a 3072-dim model), all vectors cluster at high cosine distance. No threshold value will separate them, and all prompts fall to the catch-all. The Matryoshka truncation case is a specific footgun: some models support truncation via an API parameter, but unless the Kong plugin config and the embedding model are configured to the same dimensionality, routing breaks silently. The existing dimensionality section should be extended with a direct callout, and the `ai-proxy-advanced` plugin reference should reinforce it.

**4. Description authoring guidance — missing across all docs.**
There is no guidance anywhere on how to write `description` fields for semantic routing targets. Verbose prose descriptions produce embeddings that sit further apart in the vector space, which makes threshold tuning unreliable and often causes prompts to fall to the catch-all. Descriptions should be short, concrete, and semantically distinct from each other. This should appear both in the semantic-similarity reference and in the `ai-proxy-advanced` plugin page near the `targets[].description` parameter.

**5. Fix contradictory threshold caption — bug in existing content.**
The Similarity threshold section correctly states that higher threshold = looser matching. The Figure 5 caption reads "as the similarity threshold decreases (becomes more permissive), cache hit rate increases" — which is the opposite of what the body says. One of them is wrong. Verify against the actual implementation and fix.

AHA Idea: https://konghq.aha.io/ideas/ideas/AI-I-161

Size: M

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the semantic-similarity reference and the ai-proxy-advanced plugin page, then inspect balance.lua:166 and the Redis num_docs behavior described in the issue. Verify the threshold behavior against the implementation and update the named documentation sections, Figure 5 caption, and targets[].description guidance. Done means the lifecycle, stale or empty index behavior, dimensions alignment, authoring guidance, and corrected threshold explanation are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.