Windows Python worker startup crash: Protobuf cache eviction, SDK selection, and Core Tools upgrade mitigation

> Please provide us with the following information:
> ---------------------------------------------------------------

### This issue is for: (mark with an `x`)
```
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```

**Investigation update, 2026-09-09:** the earlier conclusion that physical/non-editable package placement was the proven cause is withdrawn. Controlled reproduction identifies a Python worker / Protobuf native-module lifetime defect, plus a separate Functions SDK selection error. Physical non-editable packages can start successfully. This body supersedes the original packaging-based explanation.

**Upstream tracking:** [Azure/azure-functions-python-worker#1908](https://github.com/Azure/azure-functions-python-worker/issues/1908), including standalone reproductions, source references, candidate design, and Acceptance Criteria.

### Minimal steps to reproduce

The original symptom occurred while indexing the A2A incident-triage sample on Windows / Python 3.13 / Core Tools 4.10.0. Direct Python imports could succeed while `func start` crashed.

The issue can now be reproduced without this runtime, A2A, Durable, model credentials, or a dependency-version mismatch:

1. Use the Python 3.13 worker distributed with Core Tools 4.10.0 or 4.14.0.
2. Import its generated RPC types, retaining `proxy_worker.protos.RpcLog`.
3. Invoke the worker dependency-manager remove/re-add operation on its dependency path. This evicts public Protobuf/native modules but retains the worker RPC types.
4. Reimport `google._upb._message` from that same worker directory.
5. Read the retained `RpcLog.Error`; the Python subprocess exits with an access violation.

The upstream issue contains the complete runnable subprocess reproducer and a smaller Protobuf-only variant. Do not execute module-eviction experiments inside a live/shared worker.

An actual-host reproduction on **Core Tools 4.14.0** uses an empty Python 3.13 venv without customer Protobuf, with a minimal HTTP app that imports `google.protobuf.timestamp_pb2` and then calls `logging.error(...)` at module scope. With the worker bundle selected as the Protobuf fallback, the same crash occurs. An app that only logs, without the native-loading import, succeeds.

### Any log messages given by the failure

```text
Received load_function_metadata request from worker_init_request, script_file_name: function_app.py
Failed to start language worker process for runtime: python.
Language Worker Process exited.
Microsoft.Azure.WebJobs.Script.Grpc: py exited with code -1073741819 (0xC0000005).
Error building configuration in an external startup class.
Value cannot be null. (Parameter 'provider')
```

Captured native traceback reaches `proxy_worker.dispatcher.on_logging` when accessing `protos.RpcLog.Error`. A standalone debugger reproduction identifies a null-pointer read at address `0x18` inside `_message.pyd`; no private C symbol/source-line attribution is claimed.

A separately captured original Python error was:

```text
ImportError: cannot import name 'df_dumps' from 'azure.functions._durable_functions'
```

New Durable dependencies had resolved against the older worker-bundled Functions SDK. Attempting to log that useful Python error exposed the secondary native crash. The final `provider` message is downstream noise, not evidence that the model-provider configuration is invalid.

### Expected/desired behavior

The worker should index supported applications without native crashes. Missing or incompatible application dependencies must produce actionable Python errors rather than access violations or silently empty indexing. Applications must not need editable installation, a checkout-source injection, or a Protobuf pin solely to protect worker-internal objects.

### OS and Version?

Windows x64; Python 3.13.15. A standalone native-module reproduction also fails with Python 3.14.0; this is not a claim that the full application was qualified on 3.14 or that the issue was reproduced on Linux/macOS.

### Versions

| Component | Original | Latest-release comparison on 2026-09-09 |
| --- | --- | --- |
| Core Tools | 4.10.0, published 2026-04-29 | 4.14.0, published 2026-08-24 |
| Host | 4.1048.200.26180 | 4.1052.200.26352 |
| Proxy worker | 4.43.0 | 4.45.1 |
| V2 library worker | 1.1.0 | 1.1.1 |
| Python 3.13 bundled Protobuf | 5.29.6 | 5.29.6 |
| Python 3.13 bundled azure-functions | 1.25.0b3 | 2.2.0b5 |

The physical application dependency closure included `azure-functions-durable==2.0.0b2`, `durabletask==1.10.1`, `grpcio==1.83.1`, `protobuf==6.33.6`, and `azure-functions==2.3.0`. The original worker bundled grpcio 1.70.0.

The release chain was checked against versioned Host worker manifests **and actual Windows Core Tools distribution/startup logs**. Worker 4.46.0 exists separately but is not bundled in Core Tools 4.14.0.

### Mention any other details that might be useful

#### Root cause: established mechanism

Worker dependency isolation evicts the Protobuf Python/native modules while keeping `proxy_worker.*` and its generated message classes alive. A subsequent import can reinitialize the same native module; accessing the retained types then crashes.

Same-file, same-version reproduction succeeds without eviction and crashes with eviction/reimport for Protobuf 5.29.6 and 6.33.6 on Python 3.13, and 6.33.5 on Python 3.14. Therefore, neither mixed versions, gRPC, Durable, A2A, nor physical application-package placement is necessary to trigger the underlying fault. This does not prove every earlier startup failure had the same cause.

#### Resolution available now: upgrade improves dependency selection

Core Tools [4.14.0](https://github.com/Azure/azure-functions-core-tools/releases/tag/4.14.0) includes Python worker [#1833](https://github.com/Azure/azure-functions-python-worker/pull/1833), which prioritizes local interpreter site-packages when the customer path was not otherwise found. This avoids the old dependency-ordering problem in the observed ordinary environment.

| Observation | Outcome |
| --- | --- |
| Same minimal native-import/logging app, existing global environment, no PYTHONPATH override, Core Tools 4.10.0 | Native crash |
| Same app/environment, Core Tools 4.14.0 | HTTP 200 |
| A2A sample factory copy, Core Tools 4.14.0, existing global environment, no PYTHONPATH override | Agent Card HTTP 200 |
| A2A sample factory copy, Core Tools 4.14.0, external venv with physical non-editable dependencies, no PYTHONPATH or checkout-source injection | Agent Card HTTP 200 |
| Minimal native-import/logging app, Core Tools 4.14.0, empty venv without customer Protobuf | Native crash |

The physical-dependency comparison used a new venv whose site-packages was a junction to the previously installed physical dependency target, with bytecode writes disabled. It was not a fresh installation test. The global environment contains an editable runtime, so the separate physical run is the evidence that editable installation is not required.

**Upgrade is an effective mitigation for the observed dependency-selection problem, not a permanent fix to native-module lifetime.** These new runs established indexing and Agent Card handling, not model invocation or Durable workflow execution. Earlier model E2E success under the old workaround does not qualify the new configuration.

#### Workarounds and their limits

- Prefer a current supported Core Tools release and a compatible, complete customer dependency closure in the interpreter selected by Core Tools. Verify actual module resolution; a newer bundled SDK alone does not establish compatibility with every Durable release.
- The older explicit-customer-PYTHONPATH/editable-source setup happened to avoid the failure but should not be documented as a required package layout or a release-quality fix. Physical packages also work under controlled dependency resolution.
- Setting `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` avoided the native crash in a diagnostic control. It is process-wide, has unmeasured performance/compatibility effects, and does not independently fix `df_dumps`; it is not a qualified general workaround.
- Pinning grpcio/durabletask did not resolve the original failure. Do not ship arbitrary dependency pins or patch shared Core Tools files based on these experiments.

#### Permanent resolution: upstream design, not an implemented fix

Track the worker fix in [azure-functions-python-worker#1908](https://github.com/Azure/azure-functions-python-worker/issues/1908). The required invariant is a coherent worker-owned Protobuf runtime, native state, descriptor pools, and generated RPC types throughout their lifetime, without constraining customer public dependencies.

A candidate is a private worker Protobuf family, initially prototyped with pure-Python serialization, together with consistently rewritten generated types/imports. It must not shadow the customer `google` namespace or force process-wide implementation settings. Performance, packaging, proxy/library interoperability, and platform/lifecycle behavior require qualification.

Preserving only the native module caused Python type-identity errors in an isolated experiment. Preserving the entire public family broke newer customer generated code against the older worker runtime. Neither is a production-ready fix. Functions SDK selection is a separate compatibility track.

#### Runtime follow-up / closure criteria

- [ ] Record the first qualified upstream worker/Host/Core Tools release that removes the native failure, not merely a configuration that avoids it.
- [ ] Validate ordinary physical/wheel installation and supported startup instructions without requiring editable/source-injection workarounds.
- [ ] Exercise real A2A requests and Durable workflows against the chosen release/environment; Agent Card HTTP 200 is not sufficient release qualification.
- [ ] Confirm that missing/incompatible dependencies surface useful Python errors and do not cause silent empty indexing.
- [ ] Update sample setup/documentation to the qualified dependency-loading contract; keep the optional-Durable decoupling work separate from the upstream native fix.

For the next investigator, start with the upstream issue's minimal reproduction and Acceptance Criteria. Use a dedicated worker worktree and isolated CLI distribution; do not modify a shared Core Tools installation. No permanent worker patch or PR has been produced by this investigation.

> ---------------------------------------------------------------
> Thanks! We'll be in touch soon.

Start with Azure/azure-functions-python-worker#1908, its minimal subprocess reproduction, and Acceptance Criteria; use a dedicated worker worktree and isolated Core Tools distribution. Investigate the worker-owned Protobuf lifetime and SDK-selection tracks separately. Done means a qualified release removes the native failure, supports ordinary physical or wheel installation, exercises A2A and Durable workflows, and reports useful dependency errors.

Written by the indexing model from the issue text.