Slow event consumption and massive amount of JSONDecodeError in 26.1.0
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Self-Hosted Version
26.1.0
### CPU Architecture
arm64
### Docker Version
25.0.14
### Docker Compose Version
25.0.14
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Installation Type
upgrade from 25.12.1 to 26.1.0
### Steps to Reproduce
1. Do a clean install of 26.1.0
2. use the "restore" script to restore soft settings
3. turn on and watch system stats
### Expected Result
Sentry should run smoothly, not log errors, and overall perform well
### Actual Result
Redis rapidly fills up with events, eventually leading to a full system crash when it hits the system cap (64GB in our case on a 16-core Graviton 4 instance).
Events seem to be processed very slowly overall - there is no difference in incoming events from the client side.
Investigating the logs, we see an absolutely massive amount of the following if that may be a cause:
```
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
```
By "massive", I mean we get the above error roughly 10 times a second.
### Event ID
_No response_
Contributor guide
Research direction
Start with src/sentry/utils/safe.py and the taskworker logs around loads.process_error, then reproduce the 26.1.0 upgrade and restore sequence while watching Redis and event processing. Trace the source of the invalid input and verify that event consumption no longer produces repeated JSONDecodeError messages or allows Redis to fill and crash the system.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100