NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

Session affinity keys on an empty session id, and is silent when it cannot key at all

Open
#301 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Scope clarification

The observed 9-judge-call TB-Lite failure was specifically Harbor running Hermes with session_affinity = true and message_hash_fallback omitted/false. Hermes did not send a native session header. Harbor did generate a stable UUID per trial attempt, but sent it as proxy_x_session_id, which Switchyard consumed for routing statistics but not affinity.

If message_hash_fallback = true, affinity was already best-effort functional by hashing the first user message. PR #308 therefore adds a fourth fix that recognizes Harbor's exact per-attempt ID: it makes Harbor/Hermes sticky without the fallback and supersedes the heuristic when the fallback is enabled. The empty-ID safeguard and diagnostic changes are general rather than Harbor/Hermes-specific.

Correction to section 1 below: current main already trims empty bare HTTP header values after #269. The empty-ID libsy safeguard still protects structured metadata containing an empty nested session value and direct callers that construct Metadata { session_id: Some("") }.

Summary

Session affinity has two failure modes that are silent: it can key on an identity that
isn't one, and it can key on nothing at all. Both surface as a route that reports itself
as configured while behaving as if affinity were off -- or, in the first case, worse than
off. We hit the second while benchmarking TB-Lite and lost a while to it, then found the
first by reading the surrounding code.

1. An empty session id is treated as a real identity

AffinityRouter::affinity_key accepts whatever metadata.session_id holds:

// crates/libsy/src/algorithms/util/affinity.rs
if let Some(metadata) = request.metadata.as_ref()
    && let Some(session) = metadata.session_id.clone()
{
    ...
    return Some(AffinityKey::Session(session));
}

Header parsing does not normalise empty values — resolve_path in
crates/protocol/src/metadata.rs returns Some(raw.clone()) verbatim — so a harness that
sends x-switchyard-session-id: with an empty value yields Some(""). Every request in
every task then shares AffinityKey::Session(""), the first task's model latches, and the
whole run inherits it. The message-hash fallback cannot save this: it lives in the else
branch that is now unreachable.

This is cross-task contamination that produces plausible-looking numbers, not an error.

The sibling consumer of the same field already guards it, which is what makes this look
like an oversight rather than intent:

// crates/libsy/src/algorithms/fall_through.rs
fn session_id(request: &Request) -> Option<String> {
    request.metadata.as_ref()?
        .session_id.as_deref()
        .filter(|id| !id.is_empty())   // affinity.rs has no equivalent
        .map(str::to_string)
}

nonempty_header in crates/switchyard-server/src/routing_log.rs applies the same guard.
No existing test covers an empty session id.

2. Affinity that can never key anything says nothing

message_hash_fallback defaults to false (#[serde(default)] on a bool). So this
configuration is silently inert for any harness that sends no session header:

session_affinity = true
# message_hash_fallback not set -> false

affinity_key returns None on every request, every turn is classified afresh, and there
is no warning anywhere — warn! appears nowhere in affinity.rs or in the server's
config.rs. The behaviour is documented in docs/reference/toml_schema.md and
docs/routing_algorithms/llm_classifier_routing.md, but nothing at runtime tells you it is
happening.

Concretely: benchmark/run-baseline.sh sends no session header, so any benchmark using
session_affinity = true without the fallback silently measures per-turn routing while
reporting a sticky configuration. We measured 9 judge calls across 9 turns of a single
task before spotting it.

3. Request logs cannot distinguish absent from empty

// crates/switchyard-server/src/lib.rs
session_id = self.session_id.as_deref().unwrap_or(""),

A missing session and an empty one both render as session_id=, which is precisely the
distinction you need when diagnosing either problem above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/libsy/src/algorithms/util/affinity.rs and compare its session handling with crates/libsy/src/algorithms/fall_through.rs. Then inspect metadata resolution in crates/protocol/src/metadata.rs and request logging in crates/switchyard-server/src/lib.rs, along with the documented configuration and benchmark script. Done means empty and absent session IDs are distinguishable, affinity behavior is no longer silently inert, and the relevant diagnostics and tests cover these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.