getsentry / getsentry/self-hosted

Postgres floating-point exception but health check was ok

Open
#2,958 1 comment 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 3h
Merged PRs (30d)
12

Description

### Self-Hosted Version

24.3.0

### CPU Architecture

x86_64

### Docker Version

25.0.3

### Docker Compose Version

2.25.0

### Steps to Reproduce

You try to load any admin page like
https://sentry.domain.com/organizations/[ORGA]/projects/
And receive a HTTP Code 500

But when you call
https://sentry.domain.com/_health/
You still get an HTTP Code 200 and the message: ok

### Expected Result

When there is an error which causes all http request to fail with a HTTP Code 500, the health endpoint should also reflect this.

### Actual Result

We experienced a strange error with Sentry.
The PostgreSQL Database started to respond with an error to all SELECT set_config queries.
Web, Cron, Worker all show the same errors caused by postgres.
All API endpoints, Admin interface etc have thrown server errors and a HTTP Code 500 but /_health/ was returning HTTP 200 and a OK.
The problem lastet for 5 hours because monitoring thought the service is still alive.

All services were running.
Restart the instance and all Sentry services fixed the problem.

docker compose logs postgres:

```
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
```

docker compose logs web:
```
web-1 | psycopg2.errors.FloatingPointException: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
web-1 |
web-1 | The above exception was the direct cause of the following exception:
web-1 |
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/base.py", line 306, in handle_exception
web-1 | response = super().handle_exception(exc)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
web-1 | self.raise_uncaught_exception(exc)
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
web-1 | raise exc
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/base.py", line 411, in dispatch
web-1 | self.initial(request, *args, **kwargs)
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 312, in sentry_patched_drf_initial
web-1 | return old_drf_initial(self, request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 414, in initial
web-1 | self.perform_authentication(request)
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 324, in perform_authentication
web-1 | request.user
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 227, in user
web-1 | self._authenticate()
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 380, in _authenticate
web-1 | user_auth_tuple = authenticator.authenticate(self)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 197, in authenticate
web-1 | return self.authenticate_credentials(relay_id, relay_sig, request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 203, in authenticate_credentials
web-1 | relay, static = relay_from_id(request, relay_id)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 128, in relay_from_id
web-1 | relay = Relay.objects.get(relay_id=relay_id)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
web-1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 645, in get
web-1 | num = len(clone)
web-1 | ^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 382, in __len__
web-1 | self._fetch_all()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
web-1 | self._result_cache = list(self._iterable_class(self))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
web-1 | results = compiler.execute_sql(
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
web-1 | cursor = self.connection.cursor()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
web-1 | return self._cursor()
web-1 | ^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 40, in inner
web-1 | return func(self, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/base.py", line 107, in _cursor
web-1 | return super()._cursor()
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
web-1 | self.ensure_connection()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection
web-1 | with self.wrap_database_errors:
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
web-1 | raise dj_exc_value.with_traceback(traceback) from exc_value
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
web-1 | self.connect()
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 677, in connect
web-1 | return real_connect(self)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 258, in connect
web-1 | self.init_connection_state()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 314, in init_connection_state
web-1 | commit_tz = self.ensure_timezone()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 296, in ensure_timezone
web-1 | cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
web-1 | django.db.utils.DataError: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
web-1 | 13:03:00 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=500 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.44' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.04009842872619629 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
web-1 | 13:03:00 [ERROR] django.request: Internal Server Error: /api/0/relays/projectconfigs/ (status_code=500 request=)
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
web-1 | self.connect()
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 677, in connect
web-1 | return real_connect(self)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 258, in connect
web-1 | self.init_connection_state()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 314, in init_connection_state
web-1 | commit_tz = self.ensure_timezone()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 296, in ensure_timezone
web-1 | cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
web-1 | psycopg2.errors.FloatingPointException: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
```

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the /_health/ endpoint and compare its behavior with the PostgreSQL connection failure shown in the web logs, including the database initialization path in sentry/db/postgres/base.py. Done means the health response reflects the reported database failure instead of returning HTTP 200 while application requests return HTTP 500.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, docker, docker-compose, postgresql, python
Domain
backend, databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.