[Feature]: External / heterogeneous scheduling hooks and richer placement signals

Area

Routing and scheduling

User problem

PAIR provides a strong transport, discovery, and inference data plane for distributed local inference, but heterogeneous clusters need richer placement signals than model ownership, queue depth, and coarse GPU utilization alone.

Today, an external scheduler cannot safely influence placement for one request without changing a persistent or proxy-wide pin, which can introduce races under concurrent traffic. Nodes that own the same model can also have very different practical runtime envelopes: characterized context limits, memory requirements, full-GPU versus CPU/offload behavior, model-load time, prompt/generation throughput, and reliability.

Requests themselves vary substantially in cost based on prompt length, requested context, expected output, model size, and modality. PAIR also has useful model-state visibility, but warm-model affinity and host-wide pressure from other compute workloads are not fully represented in routing decisions. Finally, operators need explainable placement metadata without exposing prompt content.

Desired outcome

PAIR should optionally support the following while preserving its native distributed scheduler as the self-contained default and fallback:

1. Request-scoped placement hints such as a preferred node or ranked candidate list. PAIR would continue to enforce model ownership, availability, trust, and normal safety checks.
2. Per-model/per-node runtime profiles learned empirically, including characterized context sizes, VRAM/RAM requirements, GPU/offload behavior, load time, prompt TPS, generation TPS, and stability history.
3. Weighted request-cost estimation using inputs such as prompt tokens, requested context, expected output length, model size, modality, and historical model/node cost.
4. Warm-model affinity when a suitable model is already resident and resource pressure does not favor another node.
5. An optional external placement-authority hook, with automatic fallback to PAIR's native scheduler if that authority is unavailable.
6. Host-wide admission signals so external GPU/CPU/memory pressure can influence placement without PAIR needing to manage those workloads.
7. Explainable routing telemetry showing candidates, exclusions, selected node, selection reason, and failovers, without logging prompt content.
8. Optional telemetry hardening under the same cluster trust/mTLS model used by other PAIR services.

The observable result would be safer and more predictable scheduling across heterogeneous nodes, lower model load/unload churn and tail latency, and easier integration with higher-level orchestration systems while PAIR remains the inference transport and failover layer.

Guidottavio D’Assisti
guydax@gmail.com

Alternatives considered

• Continue using PAIR's native scheduler based primarily on queued/running workload and GPU pressure. This remains a valuable default and fallback, but it cannot distinguish requests or model/node combinations with radically different practical costs.
• Use the existing manual node pin. A persistent or proxy-wide pin is not sufficient for concurrent, per-request placement and may create global-state races.
• Route directly to inference engines from an external orchestrator. This would duplicate PAIR's model eligibility, discovery, mTLS transport, streaming, and failover responsibilities.
• Infer capacity only from hardware specifications. Empirically learned runtime envelopes are more representative of real behavior across models, contexts, backends, and offload modes.

Compatibility and security implications

All proposed capabilities should be optional and backward-compatible. Requests without placement hints should retain current routing behavior, and an unavailable external authority should fall back to PAIR's native scheduler.

Placement hints must remain advisory to PAIR's eligibility, availability, and trust checks; they must not bypass model ownership, pairing, mTLS, or failover safety. Unknown or unauthorized node identifiers should be rejected or ignored according to a documented policy.

Routing telemetry should exclude prompt and response content and avoid exposing unnecessary host details. If telemetry protection is extended, existing trusted-local deployments should have a documented migration path. API extensions should use optional headers or namespaced fields so existing OpenAI- and Ollama-compatible clients continue to work unchanged.

Validation approach

Maintainers could validate the outcome with automated integration tests covering:

• a request-scoped preferred node being selected only when it is reachable, trusted, and owns the requested model;
• deterministic fallback when the preferred node or external authority is unavailable;
• concurrent requests carrying different placement hints without cross-request interference;
• ranking changes driven by distinct model/node runtime profiles, weighted request costs, and warm-model state;
• host-wide admission signals excluding a pressured node and recovering when pressure clears;
• OpenAI- and Ollama-compatible requests without new hints preserving existing behavior;
• routing telemetry explaining candidates, exclusions, selection, and failovers while containing no prompt/response content;
• unauthorized or malformed hints failing safely and telemetry endpoints enforcing the configured trust model.

A small heterogeneous test matrix with the same model available on differently characterized nodes could verify improved placement and reduced cold-load churn compared with the current scheduler.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

L’issue n’identifie ni les fichiers d’implémentation ni les points d’entrée ; commencez par localiser le code de routing et de scheduling ainsi que les tests d’intégration existants. Découpez la proposition en capacités délimitées avant de modifier quoi que ce soit. Pour considérer le travail comme terminé, il faut préserver le comportement de fallback natif, isoler les indications des requêtes concurrentes, appliquer les vérifications de trust et d’eligibility, et valider le placement et la télémétrie au moyen des scénarios de clusters hétérogènes listés.