element-hq / element-hq/synapse
API endpoint `/_synapse/admin/v1/statistics/database/rooms` throwing Python error
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 607
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#15821](https://github.com/matrix-org/synapse/issues/15821).
---
### Description
When attempting to use the API endpoint `/_synapse/admin/v1/statistics/database/rooms` via `curl`, a HTTP 500 is received while Synapse throws Python errors in the log.
### Steps to reproduce
1. `curl -XGET --header "Authorization: Bearer TOKENTOKEN" http://synapse-server/_synapse/admin/v1/statistics/database/rooms`
### Homeserver
`aguiarvieira.pt`
### Synapse Version
1.86.0
### Installation Method
Docker (`matrixdotorg/synapse`)
### Database
PostgreSQL
### Workers
Multiple workers
### Platform
Oracle Arm Free with Docker
### Configuration
Presence: yes
Retention: no
Synapse modules: shared_secret_authenticator.SharedSecretAuthProvider
### Relevant log output
```shell
matrix-matrix-synapse-1 | 2023-06-23 10:33:19,794 - synapse.http.server - 133 - ERROR - GET-3694 - Failed handle request via 'LargestRoomsStatistics':
matrix-matrix-synapse-1 | Traceback (most recent call last):
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 319, in _async_render_wrapper
matrix-matrix-synapse-1 | callback_return = await self._async_render(request)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 529, in _async_render
matrix-matrix-synapse-1 | callback_return = await raw_callback_return
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/rest/admin/statistics.py", line 133, in on_GET
matrix-matrix-synapse-1 | room_sizes = await self.stats_controller.get_room_db_size_estimate()
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/stats.py", line 93, in get_room_db_size_estimate
matrix-matrix-synapse-1 | await self.stores.main.db_pool.runInteraction(
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
matrix-matrix-synapse-1 | return await delay_cancellation(_runInteraction())
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
matrix-matrix-synapse-1 | result = context.run(
matrix-matrix-synapse-1 | ^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
matrix-matrix-synapse-1 | return g.throw(self.type, self.value, self.tb)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
matrix-matrix-synapse-1 | result = await self.runWithConnection(
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
matrix-matrix-synapse-1 | return await make_deferred_yieldable(
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
matrix-matrix-synapse-1 | result = inContext.theWork() # type: ignore[attr-defined]
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
matrix-matrix-synapse-1 | inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
matrix-matrix-synapse-1 | return self.currentContext().callWithContext(ctx, func, *args, **kw)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
matrix-matrix-synapse-1 | return func(*args, **kw)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
matrix-matrix-synapse-1 | result = func(conn, *args, **kw)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
matrix-matrix-synapse-1 | return func(db_conn, *args, **kwargs)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
matrix-matrix-synapse-1 | r = func(cursor, *args, **kwargs)
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/stats.py", line 90, in get_room_db_size_estimate_txn
matrix-matrix-synapse-1 | for room_id, freq in zip(vals, freqs):
matrix-matrix-synapse-1 | ^^^^^^^^^^^^^^^^
matrix-matrix-synapse-1 | TypeError: 'NoneType' object is not iterable
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Research direction
Reproduce the failure with the curl request, then inspect synapse/rest/admin/statistics.py and synapse/storage/controllers/stats.py around get_room_db_size_estimate_txn, where the traceback shows a NoneType iteration. Trace why the database statistics query produces a missing value. Done means the endpoint no longer returns HTTP 500 and returns room statistics successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, postgresql, python
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100