NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[feature] add a native Python host driver
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
Python applications that embed Switchyard often already own the provider client, retries, deadlines, and accounting. Today, they must drive run_stream themselves, schedule each provider call, and handle callback failures, cancellation, and stream cleanup. That repeats the same orchestration in every integration and makes lifecycle mistakes easy.
Proposed solution
Add a small Python entry point:
outcome = await switchyard.libsy.drive(algorithm, request, serve, headers=headers)
drive should run the existing Rust driver and call the host-supplied async serve(call) function for each model call. The callback completes the call with respond(...) or fail(...), and can attach the served model and provider receipt. The driver should return the normal RoutingOutcome, including an accepted response and its source metadata when routing accepts one.
The host adapter stays responsible for provider-specific work. Switchyard owns the routing loop, callback scheduling, cancellation, and native stream cleanup.
Alternatives considered
- Keep asking every Python integration to consume
run_streamdirectly. This is flexible, but duplicates lifecycle code and error handling. - Add a separate wrapper for each host integration. This would repeat the same behavior and make cleanup rules differ between integrations.
- Use the standalone server or a Rust host. That does not fit applications that need an in-process Python integration with an existing client.
Scope notes
- Owning surface: Python binding, with a thin PyO3 bridge to the existing Rust driver.
- Public interface: Adds a public Python
drive(...)API and response source metadata on model calls and outcomes. - Compatibility: Existing
run_streamcallers remain supported. No server API, routing configuration, or protocol type changes are required.
Additional context
Related implementation: PR #667.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing run_stream entry point and related implementation in PR #667, then trace the Python binding and its thin PyO3 bridge to the Rust driver. Done means a public async drive(...) API invokes the host serve callback, handles callback failures, cancellation, and stream cleanup, and returns the normal RoutingOutcome with source metadata while preserving existing run_stream callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100