[Research] Evaluate GZip-kNN for request routing

Background

PR #597 proposes using GZip-kNN as a local request classifier. It compares a request with labeled examples using compression distance, then returns either a target score or an ambiguous result for a later classifier.

This may avoid some separate judge-model calls, but we do not yet have enough evaluation to know where it is a good fit for Switchyard.

There is also some product overlap with the prefill-router work. The implementations and cost profiles are different, but both use the request itself to choose a target without making a separate judge API call:

• #506 added Transformers feature extraction.
• #539 added batched prefill inference.
• #593 adds the libsy Algorithm wrapper and is currently under review.

Questions

• How well does GZip-kNN route held-out and out-of-domain requests?
• How often does it avoid a judge call at different confidence thresholds?
• What happens to end-to-end task quality when it selects the less capable target?
• What are its latency and throughput as requests and training sets grow?
• How does it compare with the prefill router using the same requests, targets, and scoring?

The integration shape also needs to be clear. #597 exposes a Rust Classifier, but not a runnable libsy Algorithm, server/TOML route, or Python entry point. The FallThrough composition shown in the PR is not available through libsy's public API, and the adapter does not plug directly into the existing judge classifier as written.

@urirosenberg, please add any evaluation or benchmarking you have already run. Would you be willing to compare it with the prefill router once the integration in #593 settles?

Tagging @nachiketb-nvidia for context on the prefill work.

This issue is for collecting the evidence first. We can decide whether to take the implementation further after that.

Start with PR #597's Rust Classifier and the integration status described for PR #593, then review the prefill-router work in PRs #506 and #539. Done means a documented comparison on held-out and out-of-domain requests covering confidence, task quality, latency, throughput, and judge-call reduction, without assuming an implementation change.