kubeflow / kubeflow/sdk

LocalProcessBackend crashes on Windows due to bash PATH resolution conflict (WSL relay vs Git Bash)

Open
#757 3 comments 0 reactions 1 assignee Claimed by @kingabdulsalamsas111-lgtm View on GitHub
kind/bug needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

When running a training job locally via LocalProcessBackend on Windows, the job crashes with a misleading error unrelated to the actual training code:

The RPC call contains a handle that differs from the declared handle type.
Error code: Bash/Service/0x8007072c

Root Cause

LocalProcessBackend` invokes the training command via subprocess using the bare string "bash":

https://github.com/kubeflow/sdk/blob/main/kubeflow/trainer/backends/localprocess/utils.py (`get_local_train_job_script`):

return "bash", "-c", command

On Windows, `"bash"` resolves to **whichever `bash.exe` is first in PATH**. It is common for Windows systems to have multiple `bash.exe` binaries:

- `C:\Windows\System32\bash.exe` — a WSL relay launcher
- `C:\Program Files\Git\usr\bin\bash.exe` — Git Bash

If the WSL relay is earlier in PATH (a common default) and the WSL distro is not in a valid/running state, the subprocess call fails immediately with the cryptic RPC error above — even if a fully working Git Bash install is present on the same machine.

Reproduction Steps

1. On Windows, ensure both WSL (in any state) and Git Bash are installed.
2. Confirm PATH order causes `bash` to resolve to the WSL relay:
(powershell)
where.exe bash
# C:\Windows\System32\bash.exe <- resolves first
# C:\Program Files\Git\usr\bin\bash.exe

3. Follow the official `examples/pytorch/image-classification/mnist.ipynb` notebook using `LocalProcessBackend`:
```python
from kubeflow.trainer import CustomTrainer, TrainerClient, LocalProcessBackendConfig

client = TrainerClient(backend_config=LocalProcessBackendConfig())
for runtime in client.list_runtimes():
if runtime.name == "torch-distributed":
torch_runtime = runtime
break

job_name = client.train(trainer=CustomTrainer(func=train_fn), runtime=torch_runtime)
for log in client.get_job_logs(job_name, follow=True):
print(log, end="")
```
4. Observe the job fails immediately with:
```
The RPC call contains a handle that differs from the declared handle type.
Error code: Bash/Service/0x8007072c
```

## Verification

Confirmed the two `bash.exe` binaries behave differently when invoked directly:

```powershell
# WSL relay - fails
> bash --version
<3>WSL (10 - Relay) ERROR: CreateProcessCommon:818: execvpe(/bin/bash) failed: No such file or directory

# Git Bash - works
> & "C:\Program Files\Git\usr\bin\bash.exe" --version
GNU bash, version 5.3.9(1)-release (x86_64-pc-cygwin)

## Expected Behavior

`LocalProcessBackend` should resolve a working shell deterministically rather than relying on unqualified PATH lookup — for example, using `shutil.which("bash")` with validation, or falling back through known-good bash locations on Windows.

## Environment

- OS: Windows
- Kubeflow SDK version: 0.4.1
- Backend: LocalProcessBackend
- Tested against: kubeflow/trainer main branch (current with v2.3.0 timeframe)

## Impact

Any Windows user following the official example notebooks with `LocalProcessBackend` who has multiple `bash.exe` installations on PATH (common with Git for Windows installed) will hit this crash, with an error message that gives no indication the actual cause is bash resolution.

### What did you expect to happen?

LocalProcessBackend should successfully start the training process on Windows when a valid Bash installation is available. If multiple Bash installations are present, it should resolve a usable Bash executable reliably or provide a clear error indicating the required Bash configuration, rather than failing with the misleading Bash/Service/0x8007072c error

### Environment

Kubernetes version:
```bash
$ kubectl version

```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow

```

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.