Azure / Azure/azure-functions-python-worker

DependencyManager purges the `google` namespace package but keeps `google.protobuf`, leaving the parent unbound (function load fails)

Open
#1,906 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
357
Forks
116
Avg merge
32m
Merged PRs (30d)
1

Description

### Summary

`DependencyManager.prioritize_customer_dependencies()` purges the `google` **namespace** package from `sys.modules` while leaving the regular subpackage `google.protobuf` cached. Any later `import google.protobuf` then takes the `sys.modules` fast path, which does not re-bind the submodule onto its parent, while `__import__` creates a fresh, empty `google` module. The result is a process where `import google.protobuf` succeeds but `google.protobuf.__version__` raises.

To be explicit about where each half lives, since the failing line is not in this repository: the **defect** is the purge in `azure_functions_worker/utils/dependency.py`, and the **symptom** surfaces in the third-party `google-api-core` package, which reads that attribute at module scope and has no reason to guard it:

```python
# google/api_core/grpc_helpers.py
29 import google.protobuf
30
31 PROTOBUF_VERSION = google.protobuf.__version__ # <-- AttributeError
```

So **every function whose import chain reaches `google.api_core` fails to load**, for the whole life of that worker process. In our case that is any function importing a Google Cloud client (Discovery Engine). Because it fails during `function_load`, no retry can clear it: an affected worker fails every invocation it is given.

### Why the purge is asymmetric

All references below are to `workers/azure_functions_worker/utils/dependency.py` at tag `4.45.1`.

[`prioritize_customer_dependencies()`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L124) re-adds the worker tree via [`_add_to_sys_path(..., add_to_first=False)`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L170) and then calls [`_remove_module_cache()`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L372), which filters cached modules by [`set(getattr(module, '__path__', None) or [])`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L401) and pops anything whose path [starts with the tree being removed](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L405-L406). The ordering is the problem:

* `google` is a namespace package. Its `__path__` is a `_NamespacePath`, whose `__iter__` re-computes against the just-mutated `sys.path`, so it re-acquires `/google`, matches the prefix test, and is popped.
* `google.protobuf` is a regular package with a static `__file__`. If it first resolved from the customer tree it does not match, and survives.

CPython then does the rest: `import a.b` with `sys.modules['a.b']` present skips both the parent import and the `setattr(parent, 'b', mod)`.

Note that `from google.cloud import storage` on an adjacent line keeps working, because `from X import Y` has a `sys.modules` fallback that plain attribute access does not. That makes this look intermittent when it is not.

### Reproduction

Deterministic (200/200 across randomised `PYTHONHASHSEED`), driving the real `DependencyManager` at tag `4.45.1`. No threads and no concurrency involved.

```dockerfile
FROM python:3.10-slim
RUN pip install --no-cache-dir --target=/trees/app \
google-cloud-discoveryengine==0.11.8 google-api-core==2.20.0 \
grpcio==1.83.0 protobuf==4.25.5
RUN pip install --no-cache-dir --target=/trees/worker \
protobuf==4.25.5 grpcio==1.83.0 azure-functions==1.21.3
RUN pip install --no-cache-dir azure-functions-worker==4.45.1 || true
WORKDIR /w
```

```python
"""Minimal repro: DependencyManager purges the `google` namespace package but keeps
`google.protobuf`, leaving the parent unbound. Run inside the image built from the Dockerfile."""
import os
import sys

# The purge below is gated on this; it is the documented setting for apps whose dependencies
# collide with the worker's, and is the default from Python 3.13.
os.environ['PYTHON_ISOLATE_WORKER_DEPENDENCIES'] = '1'

WORKER_TREE, APP_TREE = '/trees/worker', '/trees/app'

# Layout the Functions host produces: the app's .python_packages and the worker's own tree are
# both importable, and both contain part of the `google` namespace.
sys.path.insert(0, WORKER_TREE)
sys.path.insert(0, APP_TREE)

# The worker imports protobuf for its own gRPC channel before any customer code runs.
import google, google.protobuf # noqa: E401
print('before : google in sys.modules =', 'google' in sys.modules,
'| google.protobuf bound on parent =', hasattr(google, 'protobuf'))

sys.path.insert(0, '/azure-functions-worker-src/workers') # worker source checkout
from azure_functions_worker.utils.dependency import DependencyManager as DM # noqa: E402

DM.initialize()
DM.worker_deps_path = WORKER_TREE
DM.cx_deps_path = APP_TREE
DM.cx_working_dir = '/wwwroot'
DM.prioritize_customer_dependencies('/wwwroot')

print('after : google in sys.modules =', 'google' in sys.modules,
'| google.protobuf in sys.modules =', 'google.protobuf' in sys.modules)

import google.protobuf # succeeds: sys.modules fast path
print('line 29 : import google.protobuf -> OK')
try:
v = google.protobuf.__version__ # the failing read
print('line 31 : google.protobuf.__version__ ->', v)
except AttributeError as e:
print('line 31 : google.protobuf.__version__ -> AttributeError:', e)
```

Clone the worker source at the tag and mount it at `/azure-functions-worker-src`, then:

```
docker build -t afpw-repro:3.10 .
docker run --rm -v "$PWD:/w" -v ":/azure-functions-worker-src" -w /w afpw-repro:3.10 python /w/repro.py
```

Output:

```
before : google in sys.modules = True | google.protobuf bound on parent = True
after : google in sys.modules = False | google.protobuf in sys.modules = True
line 29 : import google.protobuf -> OK
line 31 : google.protobuf.__version__ -> AttributeError: module 'google' has no attribute 'protobuf'
```

The `after` line is the defect: parent evicted, child retained.

The Dockerfile pins `google-api-core` to 2.20.0 so the line numbers match the trace above. The unguarded read is not specific to that release: it is present unchanged in 2.11.0 through 2.29.0 (currently at lines 25 and 30), so upgrading the library does not avoid this.

### Impact seen in production

A Python 3.10 Linux Consumption function app, host build `4.1052.300.26370`:

* **224,951 occurrences** over seven days, 81% of all exceptions in the window
* **66.8% of invocations failed** on the affected function
* 319 to 547 distinct instances affected per day. Incidence tracked instance count almost linearly, because every cold start re-runs the specialization that triggers this.

On a staging app with the same shape, a diagnostic added at the top of every function module reported the namespace already broken in **4 of 4** cold-started worker processes. On this configuration the condition is not occasional, it is universal. What varies is only whether the module a given worker loads happens to traverse `google.api_core`.

### Suggested fix

In `_remove_module_cache()`, either purge parent and children together, or skip namespace packages whose `__path__` is dynamic. Alternatively, do the `sys.path` mutation after computing the set of modules to evict, so a `_NamespacePath` cannot re-acquire the path being removed. Leaving `sys.modules` with a child but no parent binding is the state that makes the failure possible.

### Workaround, for anyone who lands here

Re-bind the submodule before importing any Google client, as the first statement of every function module. A bare `import google.protobuf` is **not** enough, because that is precisely the statement that succeeds without binding:

```python
import importlib
import google

if not hasattr(google, 'protobuf'):
google.protobuf = importlib.import_module('google.protobuf')
```

Verified in a real environment: with this in place, cold-start bursts produce zero occurrences, and the diagnostic still shows the guard re-binding on every fresh worker. It is repairing the condition, not masking its absence.

### A question we cannot answer from outside

The purge is gated behind `PYTHON_ISOLATE_WORKER_DEPENDENCIES` in the classic worker, and both `..._DEFAULT` and `..._DEFAULT_310` are `False` in 4.45.1 and 4.46.0. Yet our production incident occurred with that app setting **absent**, where the published gating says `prioritize_customer_dependencies` should never run. A staging app shows the process-visible value matching its app setting, so we see no injection there.

Does the Linux Consumption placeholder image set `PYTHON_ISOLATE_WORKER_DEPENDENCIES` at container level, or did host hotfix `4.1052.300.26370` bundle a worker whose gating differs from the tagged sources? Worker system logs are not forwarded to Application Insights, so we cannot tell from our side.

### Environment

| | |
|---|---|
| Worker | 4.45.1 (repro), also checked 4.46.0 |
| Host | 4.1052.300.26370 |
| Python | 3.10 |
| Plan | Linux Consumption (Y1) |
| Programming model | v1 |
| Relevant packages | `grpcio` 1.83.0, `protobuf` 4.25.5, `google-api-core` 2.20.0, `google-cloud-discoveryengine` 0.11.8 |

Contributor guide

Open the contributing guide

Research direction

Start in workers/azure_functions_worker/utils/dependency.py, reading prioritize_customer_dependencies() and _remove_module_cache() around the linked lines. Run the Docker reproduction to confirm the parent/child sys.modules state, then use the existing dependency-management tests or add focused coverage; done means the purge no longer leaves google.protobuf cached without its google parent binding.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.