Azure / Azure/azure-functions-python-worker
DependencyManager purges the `google` namespace package but keeps `google.protobuf`, leaving the parent unbound (function load fails)
- Dominant language
- Python
- Stars
- 357
- Forks
- 116
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
### Summary
`DependencyManager.prioritize_customer_dependencies()` purges the `google` **namespace** package from `sys.modules` while leaving the regular subpackage `google.protobuf` cached. Any later `import google.protobuf` then takes the `sys.modules` fast path, which does not re-bind the submodule onto its parent, while `__import__` creates a fresh, empty `google` module. The result is a process where `import google.protobuf` succeeds but `google.protobuf.__version__` raises.
To be explicit about where each half lives, since the failing line is not in this repository: the **defect** is the purge in `azure_functions_worker/utils/dependency.py`, and the **symptom** surfaces in the third-party `google-api-core` package, which reads that attribute at module scope and has no reason to guard it:
```python
# google/api_core/grpc_helpers.py
29 import google.protobuf
30
31 PROTOBUF_VERSION = google.protobuf.__version__ # <-- AttributeError
```
So **every function whose import chain reaches `google.api_core` fails to load**, for the whole life of that worker process. In our case that is any function importing a Google Cloud client (Discovery Engine). Because it fails during `function_load`, no retry can clear it: an affected worker fails every invocation it is given.
### Why the purge is asymmetric
All references below are to `workers/azure_functions_worker/utils/dependency.py` at tag `4.45.1`.
[`prioritize_customer_dependencies()`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L124) re-adds the worker tree via [`_add_to_sys_path(..., add_to_first=False)`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L170) and then calls [`_remove_module_cache()`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L372), which filters cached modules by [`set(getattr(module, '__path__', None) or [])`](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L401) and pops anything whose path [starts with the tree being removed](https://github.com/Azure/azure-functions-python-worker/blob/4.45.1/workers/azure_functions_worker/utils/dependency.py#L405-L406). The ordering is the problem:
* `google` is a namespace package. Its `__path__` is a `_NamespacePath`, whose `__iter__` re-computes against the just-mutated `sys.path`, so it re-acquires `/google`, matches the prefix test, and is popped.
* `google.protobuf` is a regular package with a static `__file__`. If it first resolved from the customer tree it does not match, and survives.
CPython then does the rest: `import a.b` with `sys.modules['a.b']` present skips both the parent import and the `setattr(parent, 'b', mod)`.
Note that `from google.cloud import storage` on an adjacent line keeps working, because `from X import Y` has a `sys.modules` fallback that plain attribute access does not. That makes this look intermittent when it is not.
### Reproduction
Deterministic (200/200 across randomised `PYTHONHASHSEED`), driving the real `DependencyManager` at tag `4.45.1`. No threads and no concurrency involved.
```dockerfile
FROM python:3.10-slim
RUN pip install --no-cache-dir --target=/trees/app \
google-cloud-discoveryengine==0.11.8 google-api-core==2.20.0 \
grpcio==1.83.0 protobuf==4.25.5
RUN pip install --no-cache-dir --target=/trees/worker \
protobuf==4.25.5 grpcio==1.83.0 azure-functions==1.21.3
RUN pip install --no-cache-dir azure-functions-worker==4.45.1 || true
WORKDIR /w
```
```python
"""Minimal repro: DependencyManager purges the `google` namespace package but keeps
`google.protobuf`, leaving the parent unbound. Run inside the image built from the Dockerfile."""
import os
import sys
# The purge below is gated on this; it is the documented setting for apps whose dependencies
# collide with the worker's, and is the default from Python 3.13.
os.environ['PYTHON_ISOLATE_WORKER_DEPENDENCIES'] = '1'
WORKER_TREE, APP_TREE = '/trees/worker', '/trees/app'
# Layout the Functions host produces: the app's .python_packages and the worker's own tree are
# both importable, and both contain part of the `google` namespace.
sys.path.insert(0, WORKER_TREE)
sys.path.insert(0, APP_TREE)
# The worker imports protobuf for its own gRPC channel before any customer code runs.
import google, google.protobuf # noqa: E401
print('before : google in sys.modules =', 'google' in sys.modules,
'| google.protobuf bound on parent =', hasattr(google, 'protobuf'))
sys.path.insert(0, '/azure-functions-worker-src/workers') # worker source checkout
from azure_functions_worker.utils.dependency import DependencyManager as DM # noqa: E402
DM.initialize()
DM.worker_deps_path = WORKER_TREE
DM.cx_deps_path = APP_TREE
DM.cx_working_dir = '/wwwroot'
DM.prioritize_customer_dependencies('/wwwroot')
print('after : google in sys.modules =', 'google' in sys.modules,
'| google.protobuf in sys.modules =', 'google.protobuf' in sys.modules)
import google.protobuf # succeeds: sys.modules fast path
print('line 29 : import google.protobuf -> OK')
try:
v = google.protobuf.__version__ # the failing read
print('line 31 : google.protobuf.__version__ ->', v)
except AttributeError as e:
print('line 31 : google.protobuf.__version__ -> AttributeError:', e)
```
Clone the worker source at the tag and mount it at `/azure-functions-worker-src`, then:
```
docker build -t afpw-repro:3.10 .
docker run --rm -v "$PWD:/w" -v ":/azure-functions-worker-src" -w /w afpw-repro:3.10 python /w/repro.py
```
Output:
```
before : google in sys.modules = True | google.protobuf bound on parent = True
after : google in sys.modules = False | google.protobuf in sys.modules = True
line 29 : import google.protobuf -> OK
line 31 : google.protobuf.__version__ -> AttributeError: module 'google' has no attribute 'protobuf'
```
The `after` line is the defect: parent evicted, child retained.
The Dockerfile pins `google-api-core` to 2.20.0 so the line numbers match the trace above. The unguarded read is not specific to that release: it is present unchanged in 2.11.0 through 2.29.0 (currently at lines 25 and 30), so upgrading the library does not avoid this.
### Impact seen in production
A Python 3.10 Linux Consumption function app, host build `4.1052.300.26370`:
* **224,951 occurrences** over seven days, 81% of all exceptions in the window
* **66.8% of invocations failed** on the affected function
* 319 to 547 distinct instances affected per day. Incidence tracked instance count almost linearly, because every cold start re-runs the specialization that triggers this.
On a staging app with the same shape, a diagnostic added at the top of every function module reported the namespace already broken in **4 of 4** cold-started worker processes. On this configuration the condition is not occasional, it is universal. What varies is only whether the module a given worker loads happens to traverse `google.api_core`.
### Suggested fix
In `_remove_module_cache()`, either purge parent and children together, or skip namespace packages whose `__path__` is dynamic. Alternatively, do the `sys.path` mutation after computing the set of modules to evict, so a `_NamespacePath` cannot re-acquire the path being removed. Leaving `sys.modules` with a child but no parent binding is the state that makes the failure possible.
### Workaround, for anyone who lands here
Re-bind the submodule before importing any Google client, as the first statement of every function module. A bare `import google.protobuf` is **not** enough, because that is precisely the statement that succeeds without binding:
```python
import importlib
import google
if not hasattr(google, 'protobuf'):
google.protobuf = importlib.import_module('google.protobuf')
```
Verified in a real environment: with this in place, cold-start bursts produce zero occurrences, and the diagnostic still shows the guard re-binding on every fresh worker. It is repairing the condition, not masking its absence.
### A question we cannot answer from outside
The purge is gated behind `PYTHON_ISOLATE_WORKER_DEPENDENCIES` in the classic worker, and both `..._DEFAULT` and `..._DEFAULT_310` are `False` in 4.45.1 and 4.46.0. Yet our production incident occurred with that app setting **absent**, where the published gating says `prioritize_customer_dependencies` should never run. A staging app shows the process-visible value matching its app setting, so we see no injection there.
Does the Linux Consumption placeholder image set `PYTHON_ISOLATE_WORKER_DEPENDENCIES` at container level, or did host hotfix `4.1052.300.26370` bundle a worker whose gating differs from the tagged sources? Worker system logs are not forwarded to Application Insights, so we cannot tell from our side.
### Environment
| | |
|---|---|
| Worker | 4.45.1 (repro), also checked 4.46.0 |
| Host | 4.1052.300.26370 |
| Python | 3.10 |
| Plan | Linux Consumption (Y1) |
| Programming model | v1 |
| Relevant packages | `grpcio` 1.83.0, `protobuf` 4.25.5, `google-api-core` 2.20.0, `google-cloud-discoveryengine` 0.11.8 |
Contributor guide
Research direction
Start in workers/azure_functions_worker/utils/dependency.py, reading prioritize_customer_dependencies() and _remove_module_cache() around the linked lines. Run the Docker reproduction to confirm the parent/child sys.modules state, then use the existing dependency-management tests or add focused coverage; done means the purge no longer leaves google.protobuf cached without its google parent binding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100