NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

[feature] add a native Python host driver

Open
#683 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Problem

Python applications that embed Switchyard often already own the provider client, retries, deadlines, and accounting. Today, they must drive run_stream themselves, schedule each provider call, and handle callback failures, cancellation, and stream cleanup. That repeats the same orchestration in every integration and makes lifecycle mistakes easy.

Proposed solution

Add a small Python entry point:

outcome = await switchyard.libsy.drive(algorithm, request, serve, headers=headers)

drive should run the existing Rust driver and call the host-supplied async serve(call) function for each model call. The callback completes the call with respond(...) or fail(...), and can attach the served model and provider receipt. The driver should return the normal RoutingOutcome, including an accepted response and its source metadata when routing accepts one.

The host adapter stays responsible for provider-specific work. Switchyard owns the routing loop, callback scheduling, cancellation, and native stream cleanup.

Alternatives considered

  • Keep asking every Python integration to consume run_stream directly. This is flexible, but duplicates lifecycle code and error handling.
  • Add a separate wrapper for each host integration. This would repeat the same behavior and make cleanup rules differ between integrations.
  • Use the standalone server or a Rust host. That does not fit applications that need an in-process Python integration with an existing client.

Scope notes

  • Owning surface: Python binding, with a thin PyO3 bridge to the existing Rust driver.
  • Public interface: Adds a public Python drive(...) API and response source metadata on model calls and outcomes.
  • Compatibility: Existing run_stream callers remain supported. No server API, routing configuration, or protocol type changes are required.

Additional context

Related implementation: PR #667.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing run_stream entry point and related implementation in PR #667, then trace the Python binding and its thin PyO3 bridge to the Rust driver. Done means a public async drive(...) API invokes the host serve callback, handles callback failures, cancellation, and stream cleanup, and returns the normal RoutingOutcome with source metadata while preserving existing run_stream callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.