feat(llm-routing): support safe, tunable per-model routing configuration

Current behavior

llmConfig.routingMethod is one string in function model metadata. Invocation auth returns that string and the LLM API Gateway writes it to X-Routing-Method for the selected model. Stargate currently accepts only an algorithm name in that header and resolves a prebuilt definition from lb-config.json.

The current CLI treats commas as --llm-model and --llm-model-update field separators. The new expression therefore uses semicolons between parameters.

Decided contract

Keep one persisted field: routingMethod. It contains the selected algorithm and its direct scalar tuning values. Do not add routingParameters or another routing header.

Grammar:

<algorithm>(;<parameter>=<scalar>)*

Examples:

pulsar;seed=stable-a;consider_kv_free_tokens=true
pulsar-multiregion;seed=stable-a;n=2;max_queue_time_floor_ms=100

<algorithm> uses the current Stargate algorithm spelling, with the existing underscore-to-hyphen compatibility normalization. A scalar is one unescaped value and cannot contain a comma, semicolon, or equals sign. Commas remain reserved by the current CLI model-field grammar. A method-only value such as pulsar remains valid.

The function API and CLI perform format validation only: a non-empty method, well-formed key=value segments, valid parameter-name characters, non-empty scalar values, and no duplicate keys. They do not maintain an algorithm-specific allowlist or reject syntactically valid values based on their type, range, or applicability.

Stargate is the semantic authority. It parses scalar values according to the selected algorithm's field type, resolves the base configuration, and rejects an unknown method or parameter, an incompatible parameter, or an invalid value for the affected request before provider selection.

Required implementation

• Keep the existing function-model API, invocation-auth protobuf, and gateway request context as the one routingMethod string. Carry the persisted expression unchanged to Stargate.
• Update the CLI parser and its accepted-method list so the inline model forms support the grammar and do not reject a Stargate-supported method. Cover power-of-two, groq-multiregion, round-robin, random, pulsar, and pulsar-multiregion.
• At the LLM API Gateway boundary, always remove any inbound X-Routing-Method. Set it only from authenticated persisted model metadata. If the model has no routingMethod, forward no routing-method header and use Stargate's static default. Apply this to chat, streaming, Responses, embeddings, and proxy paths.
• In Stargate, parse the expression and resolve its method through the current configured/default/request-algorithm resolution. For an expression with parameters, clone that resolved configuration and overlay the supplied direct scalar fields. Omitted fields retain their current static value. A method-only expression must retain current behavior.
• Support every direct scalar field already represented by Stargate's load-balancer configuration. This includes request-policy fields, max_input_work_seconds, seed, and the multiregion scalar fields such as cache-affinity, queue-time, TTFT, n, max_queued, and ignore flags. Exclude the nested request_algorithms map from this compact syntax.
• Do not semantically validate values at function create or update. Persist every well-formed expression. Stargate must reject a syntactically valid but semantically invalid expression on the affected request, with a clear client error before routing.
• Parameter changes from persisted model metadata must require no ConfigMap, StatefulSet, or process reload. This requirement applies only to model-routing updates; static lb-config.json changes retain their existing lifecycle.
• Canonicalize a valid runtime configuration for identity and construct fresh routing state for a newly seen canonical configuration. In-flight requests retain their prior state; the first later request whose invocation-auth result contains the update uses the new configuration.
• Bound dynamic state per Stargate process: at most 1,024 canonical runtime definitions, LRU eviction, and 15-minute idle expiry. Bound the associated per-target load-balancer instances under the same budget so target cardinality cannot bypass the limit.
• Audit model-routing updates. Emit the full canonical routing expression in structured logs and traces after successful Stargate parsing. Keep metric labels bounded to method, outcome, and cache result; do not use expression or parameter values as metric labels.

Acceptance criteria

• A function owner can create or update one well-formed persisted routingMethod expression with direct scalar tuning values, in JSON and in both inline CLI model forms.
• The API and CLI reject malformed expressions only. They persist syntactically valid expressions even when the method, key, type, range, or algorithm compatibility is invalid.
• The gateway strips a caller-supplied routing-method header and uses only authenticated persisted metadata for every LLM request path.
• Every current Stargate routing method is accepted by the CLI. Existing method-only values preserve their current routing behavior.
• Stargate covers every supported direct scalar configuration field, rejects invalid semantic combinations before provider selection, and documents field-to-algorithm applicability and scalar types.
• Updating a model's persisted expression needs no router reload. A request that receives updated model metadata creates or uses the new canonical runtime configuration; in-flight work remains on its original state.
• Dynamic definitions and all associated per-target load-balancer instances stay within the 1,024-entry LRU budget and expire after 15 minutes idle.
• Tests cover JSON and CLI forms, format-only persistence validation, auth propagation, header stripping, static-config inheritance, semantic rejection, all algorithm/field mappings, canonical identity, state isolation, cache bounds/eviction, no-reload updates, audit events, logs, traces, and bounded metrics.
• Documentation defines the grammar, supported direct scalar fields, algorithm applicability, runtime-rejection behavior, cache policy, and telemetry behavior.