globus / globus/globus-compute

Submitter SDK on Python 3.13 fails against Python 3.12 endpoint (WorkerLost on default serializer)

Open
#2,139 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

## Summary

When a `globus-compute-sdk` submitter is on a different Python minor version
than the endpoint's worker (e.g. SDK on 3.13, worker on 3.12), tasks fail
with `WorkerLost` for any non-trivial payload. The failure mode is
inconsistent and hard to diagnose:

- Tasks that round-trip simple stdlib types (small dicts of strings) often
succeed even across the mismatch.
- Tasks that close over inline functions submitted via `Executor.submit()`
with the default serializer fail with `WorkerLost` on the worker, with a
message suggesting `AllCodeStrategies` as the workaround.
- Switching to `AllCodeStrategies` resolves the *function-serialization*
side but not the *argument/return value* pickle protocol differences. The
underlying Python `pickle` protocol diverges in places between minor
versions (stdlib type internals, some collection types), and there's no
user-facing way today to know which payloads will or won't survive.

The net effect is that a single endpoint can only reliably serve submitters
on its own Python minor version. For a research-software MCP server like
[uxarray-mcp](https://github.com/UXARRAY/uxarray-mcp-server) where the
endpoint sits on an HPC cluster (Slurm/PBS-managed conda env, often Python
3.12 because that's what the site's stack ships) and submitters are on
researcher laptops (currently Python 3.13 on fresh macOS installs), this
forces every user to pin their laptop install to match — or get cryptic
runtime errors much later in their workflow.

## Concrete reproduction

Endpoint (chrysalis at LCRC, conda env `uxarray-yac` on Python 3.12.13):

```yaml
display_name: chrysalis
engine:
type: GlobusComputeEngine
provider:
type: SlurmProvider
# ...
```

Submitter (macOS, Python 3.13.8, `globus-compute-sdk==4.12.0`):

```python
from globus_compute_sdk import Executor
from globus_compute_sdk.serialize import AllCodeStrategies, ComputeSerializer

def _ls(path):
import os
return sorted(os.listdir(path))

ex = Executor(
endpoint_id="",
serializer=ComputeSerializer(strategy_code=AllCodeStrategies()),
)
fut = ex.submit(_ls, "/lcrc/group/e3sm/.../mesh")
print(fut.result(timeout=180))
```

Result on 3.13 → 3.12: `WorkerLost: Task failure due to loss of worker N
on host chr-XXXX`. Same code on 3.12 → 3.12 returns the directory listing
in ~5s.

The warning the SDK already prints on submission is accurate:

```
Environment differences detected between local SDK and endpoint workers:
SDK: Python 3.13.8/Dill 0.3.9
Workers: Python 3.12.13/Dill 0.3.9
This may cause serialization issues.
```

— but only "may" is doing a lot of work there. In practice it's "will,
for anything beyond trivial payloads."

## Prior art on this tracker

- #2039 — SDK fails on Python 3.14+
- #1999 — Behavior diff between hosted services and source-built 3.12.6+
- #1197 — dill 0.3.5.1 / Python 3.11 / SDK 2.2.1 incompatibility

These all point at the same underlying problem: there's no published
support matrix for "which submitter Python X.Y is compatible with which
endpoint Python A.B", and the failure modes are inconsistent enough that
end users learn the constraint by hitting it in production.

## Asks (any one of these helps; in priority order)

1. **A documented compatibility matrix** for submitter Python ↔ worker
Python, even if the answer is "must match minor version." Today the
docs say "may cause serialization issues" without committing to what's
tested or supported.
2. **A pre-flight compatibility check on `Executor()` construction** that
either raises or warns *loudly* (not as a regular UserWarning that's
easy to filter) when versions don't match. The information is already
in the SDK (see #1601's text change); turn it into a structured field
downstream tools can act on.
3. **Long-term: a serialization strategy that's actually portable across
minor versions** — e.g., a documented "safe payload subset" (JSON +
numpy arrays via a stable representation) that's guaranteed to round-
trip regardless of submitter/worker Python. `AllCodeStrategies` solves
the function half; we need the data half.

## Downstream impact

We pin our package to a specific Python minor version in our docs to work
around this, because we can't tell users to follow a different version
recipe than every HPC site they connect to. Each version-pin in a
research-software package is a maintenance burden that ripples to every
user; an upstream fix or matrix would let us declare \`requires-python\`
honestly and let users mix.

Happy to provide more logs, dill/pickle traces, or run additional repro on
chrysalis if useful.

— filed from work on https://github.com/UXARRAY/uxarray-mcp-server

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.