getsentry / getsentry/self-hosted
send-beacon crashes in get_all_package_versions: dictionary changed size during iteration
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 15
Description
### Environment
self-hosted (Docker Compose)
### Version
26.9.0, upgraded from 26.8.0.
- Backend release: `26.9.0+91e940e6fe4df840c91d865b9d29df9a1b044faf`
- Python 3.13
- `sentry-sdk` 2.68.0
- `taskbroker-client` 0.20.25
### Steps to Reproduce
1. Upgrade an existing self-hosted installation from 26.8.0 to 26.9.0 with the beacon enabled.
2. Allow the hourly `sentry.tasks.send_beacon` task to run.
3. In this installation it failed at both 11:00 and 12:00 UTC on 2026-09-16, and the internal `send-beacon` cron monitor reported a failure.
I have not isolated which production module lookup changes the registry, so the upgrade steps alone may not reproduce this on every installation. The following deterministic reproduction exercises the installed helper in an isolated Python process. It simulates a module loading another module during version discovery, without modifying the real `sys.modules` or any files.
Run the following Python inside a 26.9.0 Sentry container:
```python
from pathlib import Path
from types import SimpleNamespace
import sys
source = Path("/usr/src/sentry/src/sentry/debug/utils/packages.py").read_text()
ns = {}
exec(compile(source, "packages.py", "exec"), ns)
# Isolate the reproduction from the interpreter's real module registry.
modules = {}
class LazyModule:
def __getattr__(self, key):
if key == "get_version":
# Simulate a lazy import during version lookup.
modules["loaded_during_version_lookup"] = SimpleNamespace(__version__="2.0")
return lambda: "1.0"
raise AttributeError(key)
modules["diagnostic_lazy_module"] = LazyModule()
ns["sys"] = SimpleNamespace(modules=modules, version_info=sys.version_info)
ns["get_all_package_versions"]()
```
### Expected Result
Package-version collection should tolerate modules being imported during discovery. The beacon task should not crash while constructing its payload.
### Actual Result
Production taskworker traceback:
```text
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/taskbroker_client/worker/workerchild.py", line 509, in run_worker
_execute_activation(task_func, inflight.activation, app.context_hooks)
File "/.venv/lib/python3.13/site-packages/taskbroker_client/worker/workerchild.py", line 725, in _execute_activation
task_func(*args, **kwargs)
File "/.venv/lib/python3.13/site-packages/taskbroker_client/task.py", line 142, in __call__
return self._func(*args, **kwargs)
File "/usr/src/sentry/src/sentry/tasks/beacon.py", line 165, in send_beacon
"packages": get_all_package_versions(),
File "/usr/src/sentry/src/sentry/debug/utils/packages.py", line 44, in get_all_package_versions
for module_name, app in sys.modules.items():
RuntimeError: dictionary changed size during iteration
```
The standalone reproduction fails at the same loop with the same exception.
### Analysis / Possible Fix
`get_all_package_versions()` iterates a live view of `sys.modules` and calls `get_package_version()` inside that loop. Version discovery includes attribute access and potentially calls a module's version function, so it can trigger imports. Imports from another thread can also invalidate the iterator.
A possible fix is to iterate a snapshot:
```diff
- for module_name, app in sys.modules.items():
+ for module_name, app in sys.modules.copy().items():
```
I tested this change only in memory in an isolated process: the injected lazy-module reproduction fails with the installed implementation and succeeds with the snapshot, preserving the discovered version. No production patch has been applied, and an end-to-end beacon run with the change has not been tested.
Source: [26.9.0 packages.py](https://github.com/getsentry/sentry/blob/26.9.0/src/sentry/debug/utils/packages.py#L42-L59).
Although the failure became visible after upgrading, the same unsafe loop is present in [26.8.0](https://github.com/getsentry/sentry/blob/26.8.0/src/sentry/debug/utils/packages.py#L42-L59) and was still present on `master` when checked. I am not claiming that 26.9.0 introduced it.
### Product Area
Crons
Contributor guide
Research direction
Start with src/sentry/debug/utils/packages.py at get_all_package_versions(), then run the deterministic Python reproduction from the issue inside a 26.9.0 Sentry container. Confirm package discovery tolerates modules being imported during version lookup and that the send_beacon payload can be constructed without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100