getsentry / getsentry/self-hosted
Postgres floating-point exception but health check was ok
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 12
Description
### Self-Hosted Version
24.3.0
### CPU Architecture
x86_64
### Docker Version
25.0.3
### Docker Compose Version
2.25.0
### Steps to Reproduce
You try to load any admin page like
https://sentry.domain.com/organizations/[ORGA]/projects/
And receive a HTTP Code 500
But when you call
https://sentry.domain.com/_health/
You still get an HTTP Code 200 and the message: ok
### Expected Result
When there is an error which causes all http request to fail with a HTTP Code 500, the health endpoint should also reflect this.
### Actual Result
We experienced a strange error with Sentry.
The PostgreSQL Database started to respond with an error to all SELECT set_config queries.
Web, Cron, Worker all show the same errors caused by postgres.
All API endpoints, Admin interface etc have thrown server errors and a HTTP Code 500 but /_health/ was returning HTTP 200 and a OK.
The problem lastet for 5 hours because monitoring thought the service is still alive.
All services were running.
Restart the instance and all Sentry services fixed the problem.
docker compose logs postgres:
```
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.218 UTC [763405] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.382 UTC [763406] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] ERROR: floating-point exception
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
postgres-1 | 2024-04-13 11:14:11.415 UTC [763407] STATEMENT: SELECT set_config('TimeZone', 'UTC', false)
```
docker compose logs web:
```
web-1 | psycopg2.errors.FloatingPointException: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
web-1 |
web-1 | The above exception was the direct cause of the following exception:
web-1 |
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/base.py", line 306, in handle_exception
web-1 | response = super().handle_exception(exc)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
web-1 | self.raise_uncaught_exception(exc)
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
web-1 | raise exc
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/base.py", line 411, in dispatch
web-1 | self.initial(request, *args, **kwargs)
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 312, in sentry_patched_drf_initial
web-1 | return old_drf_initial(self, request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 414, in initial
web-1 | self.perform_authentication(request)
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 324, in perform_authentication
web-1 | request.user
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 227, in user
web-1 | self._authenticate()
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 380, in _authenticate
web-1 | user_auth_tuple = authenticator.authenticate(self)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 197, in authenticate
web-1 | return self.authenticate_credentials(relay_id, relay_sig, request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 203, in authenticate_credentials
web-1 | relay, static = relay_from_id(request, relay_id)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/api/authentication.py", line 128, in relay_from_id
web-1 | relay = Relay.objects.get(relay_id=relay_id)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
web-1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 645, in get
web-1 | num = len(clone)
web-1 | ^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 382, in __len__
web-1 | self._fetch_all()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
web-1 | self._result_cache = list(self._iterable_class(self))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
web-1 | results = compiler.execute_sql(
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
web-1 | cursor = self.connection.cursor()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
web-1 | return self._cursor()
web-1 | ^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 40, in inner
web-1 | return func(self, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/base.py", line 107, in _cursor
web-1 | return super()._cursor()
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
web-1 | self.ensure_connection()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection
web-1 | with self.wrap_database_errors:
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
web-1 | raise dj_exc_value.with_traceback(traceback) from exc_value
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
web-1 | self.connect()
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 677, in connect
web-1 | return real_connect(self)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 258, in connect
web-1 | self.init_connection_state()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 314, in init_connection_state
web-1 | commit_tz = self.ensure_timezone()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 296, in ensure_timezone
web-1 | cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
web-1 | django.db.utils.DataError: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
web-1 | 13:03:00 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=500 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.44' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.04009842872619629 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
web-1 | 13:03:00 [ERROR] django.request: Internal Server Error: /api/0/relays/projectconfigs/ (status_code=500 request=)
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
web-1 | self.connect()
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 677, in connect
web-1 | return real_connect(self)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
web-1 | return func(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 258, in connect
web-1 | self.init_connection_state()
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 314, in init_connection_state
web-1 | commit_tz = self.ensure_timezone()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 296, in ensure_timezone
web-1 | cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
web-1 | psycopg2.errors.FloatingPointException: floating-point exception
web-1 | DETAIL: An invalid floating-point operation was signaled. This probably means an out-of-range result or an invalid operation, such as division by zero.
web-1 |
```
### Event ID
_No response_
Contributor guide
Research direction
Start at the /_health/ endpoint and compare its behavior with the PostgreSQL connection failure shown in the web logs, including the database initialization path in sentry/db/postgres/base.py. Done means the health response reflects the reported database failure instead of returning HTTP 200 while application requests return HTTP 500.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, docker, docker-compose, postgresql, python
- Domain
- backend, databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100