temporalio / temporalio/sdk-python

sys.monitoring callbacks (coverage, etc.) cause workflow sandbox hang on Python 3.14

Open
#1,326 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
241
Avg merge
3d 21h
Merged PRs (30d)
55

Description

hey guys I just want you to know I don't know what any of the words below mean (they were not in the bible)

the reason I'm opening this is because the pydantic-ai test suite was hanging in our PR to support python 3.14 https://github.com/pydantic/pydantic-ai/pull/4138

the hang came from the temporal tests, so me and opus started investigating and though it was very difficult we were finally I able to produce an MRE

so what you'll read (if you read it) below is opus' explanation, I'll put up a solution forward in a moment that I verified worked using the MRE and also running the test again with the patched temporal sdk


summary

on python 3.14, coverage switched from sys.settrace to sys.monitoring (PEP 669). sys.monitoring callbacks are global — they fire on all threads, including the workflow thread pool thread where the sandbox importer is active.

coverage's branch callback (sysmon_branch_either) lazily imports coverage.python the first time a BRANCH event fires. when this happens inside the sandbox, the import chain hits platform.python_implementation() which is restricted → RestrictedWorkflowAccessError → workflow task fails repeatedly → process hangs.

causal chain
greet() → `if name:` (BRANCH event)
→ coverage.sysmon.sysmon_branch_either
→ coverage.sysmon.get_multiline_map
→ coverage.parser.PythonParser.__init__
→ `from coverage.python import get_python_source`  ← lazy import
→ sandbox intercepts (coverage not in passthrough list)
→ coverage.__init__ → coverage.control → coverage.env
→ `platform.python_implementation()` ← restricted in sandbox
→ RestrictedWorkflowAccessError

this affects any sys.monitoring-based tool, not just coverage. any tool that registers monitoring callbacks and triggers a lazy import can hit this.

MRE

two files — the workflow must be in a separate module (not __main__) so the sandbox imports it normally and the branch callback fires during workflow execution rather than during import.

mre_workflow.py

from temporalio import workflow


def greet(name: str) -> str:
    if name:
        return f"Hello, {name}!"
    return "Hello!"


@workflow.defn
class SimpleWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return greet(name)

mre_coverage.py

import asyncio
import sys

from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

from mre_workflow import SimpleWorkflow


async def main() -> None:
    async with await WorkflowEnvironment.start_local() as env:
        task_queue = "mre-task-queue"
        async with Worker(
            env.client,
            task_queue=task_queue,
            workflows=[SimpleWorkflow],
        ):
            result = await asyncio.wait_for(
                env.client.execute_workflow(
                    SimpleWorkflow.run,
                    "world",
                    id="mre-workflow",
                    task_queue=task_queue,
                ),
                timeout=10,
            )
            print(f"Result: {result}")
            assert result == "Hello, world!", f"Unexpected: {result}"
            print("OK")


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except (asyncio.TimeoutError, TimeoutError):
        print(
            f"HANG DETECTED: workflow timed out (Python {sys.version_info.major}.{sys.version_info.minor})",
            file=sys.stderr,
        )
        sys.exit(1)
setup & reproduce
uv venv --python 3.14 .venv314
uv pip install --python .venv314/bin/python "temporalio[testing]" coverage

# hangs:
.venv314/bin/coverage run --source=. --branch mre_coverage.py

# works (no coverage):
.venv314/bin/python mre_coverage.py

# works (3.13 + coverage):
# uv venv --python 3.13 .venv313
# uv pip install --python .venv313/bin/python "temporalio[testing]" coverage
# .venv313/bin/coverage run --source=. --branch mre_coverage.py
why previous attempts at reproducing failed
  • workflow with no branching code → branch callback never fires
  • workflow defined in __main__ → sandbox re-imports it as __temporal_main__, and coverage's branch callback fires during import (before sandbox activation) rather than during workflow execution
environment
  • python 3.14.2
  • temporalio 1.20.0
  • coverage 7.13.4
suggested fix

suspend sys.monitoring events inside Importer.applied(). save and clear all events on enter, restore on exit. this prevents any monitoring callback from firing inside the sandbox regardless of which tool registered it.

this is better than adding coverage to passthrough because:

  • it's general — fixes all sys.monitoring-based tools
  • monitoring callbacks are non-deterministic external state that shouldn't fire inside the sandbox anyway
  • no user configuration needed

I'll put up a PR with this approach.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sandbox importer entry point named in the issue, Importer.applied(), and reproduce the two-file MRE on Python 3.14 with coverage. Trace the monitoring state while the workflow executes, then verify that the MRE completes instead of hanging and that the relevant SDK tests pass. The issue notes that a pull request is expected, so check for overlapping work before starting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.