[Bug] Python 3.13 Windows proxy worker can crash after evicting and reimporting live Protobuf native modules

## Summary

On Windows, the Python 3.13 proxy worker can exit with `0xC0000005` during indexing/logging after dependency isolation evicts Protobuf modules while retaining worker-generated RPC message classes. Reimporting the **same** `google._upb._message` binary and then accessing a retained generated type is sufficient to reproduce a native access violation. Different Protobuf versions, A2A, Durable, gRPC, and the Functions host are not required for the smallest reproduction.

Originally investigated through Azure/azure-functions-agents-runtime#211. The application's earlier physical-versus-editable packaging explanation was superseded by controlled reproductions. This issue tracks the worker/native-module lifetime defect, not an A2A implementation bug.

**Still reproducible in the actual host bundled with Core Tools 4.14.0**, using an empty Python 3.13 venv without customer Protobuf. Upgrading does improve a different dependency-resolution path; see below.

## Environment and release chain

| Component | Original | Latest Core Tools comparison (2026-09-09) |
| --- | --- | --- |
| Core Tools | 4.10.0 | 4.14.0 |
| Host | 4.1048.200.26180 | 4.1052.200.26352 |
| Proxy worker | 4.43.0 | 4.45.1 |
| V2 library worker | 1.1.0 | 1.1.1 |
| Python | 3.13.15, Windows x64 | Same |
| Bundled Protobuf for Python 3.13 | 5.29.6 | 5.29.6 |
| Bundled azure-functions for Python 3.13 | 1.25.0b3 | 2.2.0b5 |

The host versions' `eng/build/Workers.Python.props` specify the worker releases. Because that package reference excludes Windows, the actual downloaded Windows Core Tools distribution and startup logs were also inspected. Worker 4.46.0 is a separate release, not the worker bundled in Core Tools 4.14.0; a 4.46.0 host comparison has not been performed.

## Observed failure chain

1. The proxy worker imports its own Protobuf runtime and `proxy_worker.protos`, retaining generated RPC message classes.
2. `DependencyManager.prioritize_customer_dependencies()` removes/re-adds the worker dependency path. `_remove_module_cache()` exempts `proxy_worker.*`, but evicts `google.protobuf.*` and `google._upb._message` resolved from that path.
3. Customer import of `google.protobuf.timestamp_pb2` (or the native module itself) can reload the same worker `_message.pyd` if no separate customer Protobuf wins resolution.
4. The worker still holds its original RPC types. `Dispatcher.on_logging()` reads `protos.RpcLog.Error` and the process crashes inside `_message.pyd`.
5. Host reports the child process failure. Its subsequent `Value cannot be null. (Parameter 'provider')` is secondary, not evidence of a model-provider configuration error.

Relevant tagged sources: [dependency.py](https://github.com/Azure/azure-functions-python-worker/blob/cb778e282a6df422e7b3ac5817a12ea80a40347f/workers/proxy_worker/utils/dependency.py), [dispatcher.py](https://github.com/Azure/azure-functions-python-worker/blob/cb778e282a6df422e7b3ac5817a12ea80a40347f/workers/proxy_worker/dispatcher.py).

The original application also encountered `ImportError: cannot import name 'df_dumps' from 'azure.functions._durable_functions'`: new Durable dependencies were resolved against an older worker SDK. Logging that ordinary Python exception exposed the secondary native crash. **SDK selection and native-module lifetime are separate problems.**

## Reproduction A: actual worker cache transition, without host

Run each case in a disposable subprocess: the failing case intentionally terminates Python. Do not run this in a live worker or shared interpreter. No installed package modifications are required.

Save as `worker_cache_repro.py`:

```python
import argparse
import faulthandler
import importlib
import sys

parser = argparse.ArgumentParser()
parser.add_argument("worker_directory")
parser.add_argument("--clear", action="store_true")
args = parser.parse_args()
faulthandler.enable(all_threads=True)
sys.path.insert(0, args.worker_directory)

from proxy_worker import protos
from proxy_worker.utils.dependency import DependencyManager

print("Retained RpcLog class:", protos.RpcLog, flush=True)
print("Native before:", sys.modules["google._upb._message"].__file__, flush=True)
if args.clear:
DependencyManager._remove_from_sys_path(args.worker_directory)
DependencyManager._add_to_sys_path(args.worker_directory, True)
print("Native evicted:", "google._upb._message" not in sys.modules, flush=True)

importlib.import_module("google")
module = importlib.import_module("google._upb._message")
print("Native after:", module.__file__, flush=True)
print("Reading retained RpcLog.Error", flush=True)
print("RpcLog.Error =", protos.RpcLog.Error, flush=True)
```

PowerShell, using the Python 3.13 bundle from an extracted Core Tools ZIP:

```powershell
$worker = 'C:\path\to\core-tools\workers\python\3.13\WINDOWS\X64'
python -I .\worker_cache_repro.py $worker
# Control: RpcLog.Error = 4; exit 0
python -I .\worker_cache_repro.py $worker --clear
$LASTEXITCODE
# Failing released worker: -1073741819 (0xC0000005)
```

On both inspected worker releases, the failing case reports native module eviction and the same native file before/after, then faults when reading the enum.

## Reproduction B: reduce further to Protobuf only

In the same isolated Python process setup, prepend a directory containing one tested Protobuf distribution to `sys.path`, then execute:

```python
import faulthandler
import importlib
import sys
from google.protobuf import descriptor_pb2

faulthandler.enable(all_threads=True)
message_type = descriptor_pb2.FieldDescriptorProto
print(sys.modules["google._upb._message"].__file__, flush=True)
del sys.modules["google._upb._message"]
importlib.import_module("google._upb._message")
print(message_type.TYPE_DOUBLE, flush=True)
```

Omitting the delete/reimport yields `1`. The failing variant also reproduces with `python -I` and while keeping an extra strong reference to the original native module.

| Python | Protobuf | Control | Delete/reimport |
| --- | --- | --- | --- |
| 3.13.15 | 5.29.6 | Exit 0 | Native access violation |
| 3.13.15 | 6.33.6 | Exit 0 | Native access violation |
| 3.14.0 | 6.33.5 | Exit 0 | Native access violation |

The Python 3.14 row is a standalone native reproduction, not a claim that every 3.14 Functions app crashes. It also does not imply arbitrary native-module unloading must be supported by Protobuf; the worker must avoid invalidating its own live objects regardless.

## Actual host observations and controls

A minimal HTTP application imports `google.protobuf.timestamp_pb2` at module scope and calls `logging.error(...)` before defining its route. No A2A, Durable, model connection, or customer Protobuf installation is needed in the empty-venv case.

| Scenario | Result |
| --- | --- |
| 4.10.0, ordinary global interpreter, no PYTHONPATH override | Native crash; Protobuf resolves to worker bundle |
| 4.14.0, same app/interpreter environment | HTTP 200; Protobuf resolves to global customer site-packages |
| 4.14.0, empty venv, no customer Protobuf or PYTHONPATH override | Native crash; Protobuf resolves to worker bundle |
| Earlier baseline app: logging without native-loading import | Succeeds (two runs) |
| Earlier timestamp import + logging | Crashes (two runs) |
| Earlier plain `import google.protobuf` + logging | Succeeds |
| Earlier `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` control | No native crash |

For the empty-venv case, create a venv with `python -m venv --without-pip`, activate/select its interpreter, and point Core Tools at a minimal HTTP app. Keep `PYTHONPATH` and `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` unset. Confirm the selected interpreter and module paths in diagnostics; merely having a venv directory is insufficient.

A native debugger identified a null-pointer read at address `0x18` inside `_message.pyd` for the 5.29.6 reproduction. No private symbols were available, so no exact internal C function/source line is asserted.

## Upgrade / workaround status

- Core Tools 4.14.0 contains #1833, which prioritizes local interpreter site-packages when the customer path was not otherwise found. This avoids the original symptom when a compatible customer dependency closure exists. It does **not** fix the empty-venv native fallback case.
- With 4.14.0, the A2A application factory returned Agent Card HTTP 200 both in the existing global environment and in a separate external venv exposing existing **physical non-editable** dependencies, without PYTHONPATH/source injection. The latter used a directory junction to an existing package target, not a fresh pip-install run. No model call or Durable workflow execution was performed in this comparison.
- Selecting pure-Python Protobuf avoided the native crash in a diagnostic control, but is process-wide, has unmeasured performance/compatibility impact, and does not independently resolve the SDK mismatch. Treat as a temporary diagnostic option, not a qualified application fix.
- Do not patch shared Core Tools files or ship arbitrary dependency pins as the solution.

## Candidate permanent fix (proposal, not validated implementation)

Maintain one coherent worker-owned Protobuf family for the entire lifetime of worker RPC objects while allowing customer packages to resolve independently.

A candidate to prototype is a **private, worker-owned pure-Python Protobuf runtime plus generated RPC/well-known types** under the proxy worker namespace. Rewrite static/dynamic imports consistently, choose the implementation only inside that private family, and keep it out of customer cache eviction. Preserve the same family across the proxy-to-library `protos` interface, RPC serialization, descriptors, and callbacks. Do not alias it into public `google.protobuf`, shadow the `google` namespace, or force the customer's implementation via a process-wide environment variable.

Measure RPC serialization throughput, large-payload latency, CPU, and memory before approving pure Python. Performance budgets require maintainer agreement. If unsuitable, consider genuinely independent native-module state/build isolation or a larger process boundary. Renaming/copying a `.pyd` alone does not prove isolation.

Prior art #1873/#1876 applies to Python <=3.12, not this 3.13 proxy path. In particular, public aliases/process-wide settings in that fallback should not be copied blindly into a customer-independent design.

Experiments in an isolated worker copy explain why smaller patches are insufficient: preserving only the native module produced Python Message identity errors; preserving the entire public Protobuf family avoided the crash but broke customer Protobuf 6 generated code against worker runtime 5.29.6. Conditional preservation based on file existence was diagnostic only, not a portable fix.

Related #1906 concerns a different asymmetric namespace eviction/AttributeError in the classic worker. Regression coverage should include that namespace integrity constraint; purging an entire family is not safe if live worker native types remain.

## Acceptance Criteria

- [ ] A released-worker regression demonstrates the native fault in a bounded subprocess/actual host, without A2A, Durable, cloud credentials, or editable imports.
- [ ] On the corrected worker, dependency switching never invalidates the Protobuf family backing retained worker RPC objects. Indexing, logging, enum/descriptor access, and RPC serialization succeed in the formerly failing host scenario. Tests should assert the lifetime invariant, not require a particular vendoring mechanism.
- [ ] Cover no customer Protobuf, same-version customer Protobuf, newer customer runtime/gencode, and explicitly selected customer pure-Python implementation. A base HTTP app must not need to install Protobuf just to prevent a worker crash.
- [ ] Customer public package resolution and native implementation remain independent; no Message identity/descriptor-pool/version errors, and no `google.cloud`/`google.api_core` namespace regression.
- [ ] App-local packages, external venvs, explicit customer paths, startup, specialization, and reload are covered across supported Python versions/platforms. Windows 3.13 is mandatory for this regression; do not claim Linux/macOS reproduction until tested.
- [ ] Ordinary application import/configuration errors are reported with useful diagnostics instead of native exit or silent empty indexing. Track incompatible Functions SDK selection separately.
- [ ] Proxy/library integration, base HTTP, physical-wheel A2A startup/request execution, and Durable workflow behavior are exercised; preserve the host wire contract.
- [ ] Benchmark the chosen implementation and meet maintainer-agreed resource/latency budgets before release.
- [ ] Validate packaged worker artifacts and the consuming Host/Core Tools release, not just an editable checkout.

The standalone Protobuf-only delete/reimport reproducer may remain an upstream Protobuf concern even if the worker stops taking that unsafe path. Fixing the worker does not require promising arbitrary customer `sys.modules` manipulation is safe.

## Suggested handoff sequence

1. Reproduce against the exact 4.45.1 bundle and read the tagged dependency-manager/proxy-library interfaces.
2. Add focused subprocess regressions; retain the original reproducer as the failing baseline.
3. Review/prototype coherent private isolation in a dedicated worker worktree, not by modifying the shared CLI installation.
4. Resolve SDK selection as a separate track, then expand lifecycle/platform/performance coverage.
5. Coordinate release integration and update Azure/azure-functions-agents-runtime#211 with the first qualified fixed release.

No permanent worker fix has been implemented or benchmarked yet.

Start by running the bounded subprocess reproductions described in the issue, then inspect workers/proxy_worker/utils/dependency.py and workers/proxy_worker/dispatcher.py. Use the retained RPC type and Protobuf module lifetime as the starting point, and add regression coverage for the listed customer-Protobuf scenarios. Done means dependency switching no longer invalidates retained worker RPC objects and the formerly failing indexing, logging, descriptor access, and serialization paths succeed.

Scritto dal modello di indicizzazione a partire dal testo della issue.