getsentry / getsentry/sentry

Slow event consumption and massive amount of JSONDecodeError in 26.1.0

Open
#107,594 4 comments 0 reactions 0 assignees View on GitHub
Bug Self-hosted Waiting for: Product Owner
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Self-Hosted Version

26.1.0

### CPU Architecture

arm64

### Docker Version

25.0.14

### Docker Compose Version

25.0.14

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

upgrade from 25.12.1 to 26.1.0

### Steps to Reproduce

1. Do a clean install of 26.1.0
2. use the "restore" script to restore soft settings
3. turn on and watch system stats

### Expected Result

Sentry should run smoothly, not log errors, and overall perform well

### Actual Result

Redis rapidly fills up with events, eventually leading to a full system crash when it hits the system cap (64GB in our case on a 16-core Graviton 4 instance).

Events seem to be processed very slowly overall - there is no difference in incoming events from the client side.

Investigating the logs, we see an absolutely massive amount of the following if that may be a cause:

```
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
taskworker-1 | File "/usr/src/sentry/src/sentry/utils/safe.py", line 20, in safe_execute
taskworker-1 | result = func(*args, **kwargs)
taskworker-1 | orjson.JSONDecodeError: Input must be bytes, bytearray, memoryview, or str: line 1 column 1 (char 0)
taskworker-1 | 11:18:19 [ERROR] sentry.safe.builtin_function_or_method: loads.process_error
taskworker-1 | Traceback (most recent call last):
```

By "massive", I mean we get the above error roughly 10 times a second.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with src/sentry/utils/safe.py and the taskworker logs around loads.process_error, then reproduce the 26.1.0 upgrade and restore sequence while watching Redis and event processing. Trace the source of the invalid input and verify that event consumption no longer produces repeated JSONDecodeError messages or allows Redis to fill and crash the system.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.