NVIDIA / NVIDIA/OpenShell

feat(server): implement grpc.health.v1.Health/Watch streaming RPC

Open
#2,569 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:triage-needed
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Problem Statement

Clients that need to observe gateway health changes (e.g., Kaiden, potential Podman Desktop extensions, dashboards, operators) currently have no event-driven mechanism. The only options are:

  • openshell status -o json — polls the unary Health RPC, returns connected/disconnected
  • openshell gateway info -o json — polls GetGatewayInfo, returns healthy/degraded/unhealthy
  • HTTP /readyz — poll-based, and disabled by default for local gateways

All of these require polling. For integrations that need to react to gateway state transitions (healthy → degraded, connected → disconnected), polling adds latency and unnecessary load.

The standard gRPC health checking protocol defines a Watch streaming RPC for exactly this purpose.

Proposed Design

Implement the standard grpc.health.v1.Health/Watch server-streaming RPC using tonic-health.

What's already in place:

  • The /grpc.health. path prefix is already in the unauthenticated bypass list (crates/openshell-server/src/auth/oidc.rs:33), so no auth changes are needed.
  • The ServiceStatus enum (Healthy, Degraded, Unhealthy) already exists in the proto and SDK.
  • The DatabaseHealthMonitor in crates/openshell-server/src/readiness.rs already maintains a tokio::sync::watch channel with health state — this can drive the tonic-health reporter.

Implementation outline:

  1. Add tonic-health dependency to openshell-server.
  2. Register the grpc.health.v1.Health service on the gRPC server alongside the existing OpenShell service.
  3. Wire the DatabaseHealthMonitor's watch channel to the tonic-health reporter so that Watch streams reflect actual readiness state (not just "up = healthy").
  4. Expose the Watch RPC in the SDK client as a streaming health subscription.
  5. Optionally surface it in the CLI as openshell status --watch or openshell status --follow.

Behavior:

  • Watch("") (empty service) returns overall gateway health.
  • Status transitions (ServingNotServing) are pushed to subscribers when the database health monitor detects a change.
  • Existing unary Health and GetGatewayInfo RPCs remain unchanged.

Alternatives Considered

  • Polling openshell status — works but adds latency (seconds) and CPU cost for tight polling intervals. Acceptable as a fallback but not ideal for reactive integrations.
  • WebSocket health endpoint — non-standard, requires a separate HTTP upgrade path, harder for gRPC-native clients to consume.
  • Custom streaming RPC on the OpenShell service — unnecessary when the standard grpc.health.v1 protocol already exists and tooling supports it.

Agent Investigation

  • Confirmed the grpc.health.v1.Health/Watch RPC is not implemented — no tonic-health dependency exists in the project.
  • Confirmed the auth bypass for /grpc.health. is already configured at crates/openshell-server/src/auth/oidc.rs:33 with a test at line 416.
  • Confirmed the DatabaseHealthMonitor (crates/openshell-server/src/readiness.rs) already maintains real-time health state via tokio::sync::watch that could drive the reporter.
  • The current unary Health handler (crates/openshell-server/src/grpc/mod.rs:224-232) unconditionally returns Healthy — this proposal would also improve that by connecting it to actual readiness state.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/openshell-server/src/grpc/mod.rs:224-232 and crates/openshell-server/src/readiness.rs, then inspect the existing auth bypass in crates/openshell-server/src/auth/oidc.rs and its test around line 416. Trace how the DatabaseHealthMonitor watch channel exposes readiness and how the SDK client is structured. Done means the standard Health/Watch stream reports gateway transitions, the existing unary behavior remains supported, and the relevant auth and streaming tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.