getsentry / getsentry/self-hosted

We had a spike in errors and after that 100% of errors are getting dropped, could someone help me figure out why?

Open
#2,960 10 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 3h
Merged PRs (30d)
12

Description

### Self-Hosted Version

24.3.0 unknown

### CPU Architecture

x86_64

### Docker Version

24.0.7

### Docker Compose Version

2.21.0

### Steps to Reproduce

On April 8th (Monday) we experienced a spike in errors dropped. There was nothing peculiar going on this day, we didn't receive any complaints of downtime for our web application.

![image](https://github.com/getsentry/self-hosted/assets/12093243/507a3207-a541-4ff4-8dc0-f595431e50d8)

![image](https://github.com/getsentry/self-hosted/assets/12093243/9a93d42e-8c5b-4706-a77c-d2f0be5c4d44)

According to the stats page this started at 9am and from April 8th at 9am until today 100% of errors have been dropped.

I have rate limiting set up but that doesnt seem to be the cause as can be seen in screenshots below.

I don't see any warnings in the System Warnings page in the admin panel.

Anybody have any suggestions?

I'd love if Sentry showed a reason as to why the errors were dropped.

### Expected Result

Expected errors to not be dropped.

### Actual Result

Docker compose logs:
https://pastebin.com/raw/TXHJL7i3

![image](https://github.com/getsentry/self-hosted/assets/12093243/ccb07943-a326-4848-9f37-e8c0306bcd8a)
![image](https://github.com/getsentry/self-hosted/assets/12093243/c84e7bd0-8d0d-4305-b7c2-b0a02b63d9f9)
![image](https://github.com/getsentry/self-hosted/assets/12093243/e1c98104-86a1-4073-a14e-4ef604a17693)
![image](https://github.com/getsentry/self-hosted/assets/12093243/5cdddc0e-fdaa-43f2-9198-7c488c87d7cb)

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the linked Docker Compose logs and the System Warnings page, then compare the stats page with the rate-limiting configuration and the reported April 8 timeline. Done means identifying why all errors are being dropped and documenting or verifying a fix that restores error ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, shell
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.