element-hq / element-hq/synapse
/admin/v1/send_server_notice is rate limited despite having corresponding ratelimit_override entries
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#15974](https://github.com/matrix-org/synapse/issues/15974).
---
### Description
In homeserver.yaml:
```
server_notices:
system_mxid_localpart: system
```
In ratelimit_override (put both here for testing):
```
user_id | messages_per_second | burst_count
----------------------------+---------------------+-------------
@root:ourmatrixdomain.net | 0 | 0
@system:ourmatrixdomain.net | 0 | 0
```
Our application, using user `root`, after sending several messages successfully:
```
2023-07-19 14:34:55 Running task now
2023-07-19 14:34:55 The following request was sent to ourmatrixdomain.net
URL: http://worker-reverse-proxy/_synapse/admin/v1/send_server_notice/@user:ourmatrixdomain.net.ns64b7d814ada6f8.93327386
Method: PUT
Body: {"user_id":"@user:ourmatrixdomain.net","content":{"msgtype":"m.text","format":"org.matrix.custom.html","formatted_body":"Message from root<\/strong>
\nTest message<\/strong>","body":"Message from root\r\nTest message"}}
2023-07-19 14:34:55 The following response was received from ourmatrixdomain.net:
Status Code: 429
Headers: server: nginx; date: Wed, 19 Jul 2023 12:34:55 GMT; content-type: application/json; transfer-encoding: chunked; connection: keep-alive; cache-control: no-cache, no-store, must-revalidate; access-control-allow-origin: *; access-control-allow-methods: GET, HEAD, POST, PUT, DELETE, OPTIONS; access-control-allow-headers: X-Requested-With, Content-Type, Authorization, Date; access-control-expose-headers: Synapse-Trace-Id
Body: {"errcode":"M_LIMIT_EXCEEDED","error":"Too Many Requests","retry_after_ms":2150}
2023-07-19 14:34:55 Exception thrown
2023-07-19 14:34:55 Operation 'Admin\ServerNotice' failed: M_LIMIT_EXCEEDED - Too Many Requests
```
### Homeserver
Non-federating private instance
### Synapse Version
1.85.2
### Installation Method
Other (please mention below)
### Database
Single PostgreSQL 15.3
### Workers
Multiple workers
### Platform
Docker containers installed via Matrix Ansible Playbook on Ubuntu 22.04
### Relevant log output
```shell
Jul 19 12:34:51 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:51,682 - synapse.server_notices.server_notices_manager - 166 - INFO - PUT-5615 - Creating server notices room for @test30:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,015 - synapse.http.client - 924 - INFO - PUT-5615 - Received response to POST synapse-replication://matrix-synapse-worker-stream-writer-3-account-data/_synapse/replication/add_tag/%40test30%3Aourmatrixdomain.net/%21rKRbFEDCorEwFePxfq%3Aourmatrixdomain.net/m.server_notice: 200
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,022 - synapse.server_notices.server_notices_manager - 199 - INFO - PUT-5615 - Created server notices room !rKRbFEDCorEwFePxfq:ourmatrixdomain.net for @test30:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,027 - synapse.http.server - 114 - INFO - PUT-5615 - SynapseError: 429 - Too Many Requests
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,028 - synapse.access.http.8008 - 461 - INFO - PUT-5615 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.353sec/0.001sec (0.057sec, 0.014sec) (0.020sec/0.074sec/48) 80B 429 "PUT /_synapse/admin/v1/send_server_notice/@test30:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [2 dbevts]
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,099 - synapse.server_notices.server_notices_manager - 152 - INFO - PUT-5618 - Using existing server notices room !ENwElmoWoVSqgGzEDD:ourmatrixdomain.net for user @test3:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,108 - synapse.server_notices.server_notices_manager - 76 - INFO - PUT-5618 - Sending server notice to @test3:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,181 - synapse.access.http.8008 - 461 - INFO - PUT-5618 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.096sec/0.001sec (0.012sec, 0.004sec) (0.008sec/0.028sec/24) 59B 200 "PUT /_synapse/admin/v1/send_server_notice/@test3:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [3 dbevts]
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,203 - synapse.server_notices.server_notices_manager - 152 - INFO - PUT-5619 - Using existing server notices room !EuvbdfIWKOZUNEnmsa:ourmatrixdomain.net for user @test4:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,213 - synapse.server_notices.server_notices_manager - 76 - INFO - PUT-5619 - Sending server notice to @test4:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,282 - synapse.access.http.8008 - 461 - INFO - PUT-5619 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.092sec/0.001sec (0.010sec, 0.004sec) (0.008sec/0.029sec/24) 59B 200 "PUT /_synapse/admin/v1/send_server_notice/@test4:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [3 dbevts]
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,301 - synapse.server_notices.server_notices_manager - 152 - INFO - PUT-5620 - Using existing server notices room !TsKknhuNwXktoPYxbm:ourmatrixdomain.net for user @test5:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,311 - synapse.server_notices.server_notices_manager - 76 - INFO - PUT-5620 - Sending server notice to @test5:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,378 - synapse.access.http.8008 - 461 - INFO - PUT-5620 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.087sec/0.001sec (0.008sec, 0.004sec) (0.007sec/0.028sec/24) 59B 200 "PUT /_synapse/admin/v1/send_server_notice/@test5:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [3 dbevts]
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,397 - synapse.server_notices.server_notices_manager - 166 - INFO - PUT-5621 - Creating server notices room for @test6:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,733 - synapse.http.client - 924 - INFO - PUT-5621 - Received response to POST synapse-replication://matrix-synapse-worker-stream-writer-3-account-data/_synapse/replication/add_tag/%40test6%3Aourmatrixdomain.net/%21ZLYxxsMLeIntUnvUly%3Aourmatrixdomain.net/m.server_notice: 200
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,738 - synapse.server_notices.server_notices_manager - 199 - INFO - PUT-5621 - Created server notices room !ZLYxxsMLeIntUnvUly:ourmatrixdomain.net for @test6:ourmatrixdomain.net
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,743 - synapse.http.server - 114 - INFO - PUT-5621 - SynapseError: 429 - Too Many Requests
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,744 - synapse.access.http.8008 - 461 - INFO - PUT-5621 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.358sec/0.001sec (0.055sec, 0.005sec) (0.019sec/0.079sec/50) 80B 429 "PUT /_synapse/admin/v1/send_server_notice/@test6:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [2 dbevts]
Jul 19 12:34:52 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:52,780 - synapse.server_notices.server_notices_manager - 166 - INFO - PUT-5624 - Creating server notices room for @test7:ourmatrixdomain.net
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,037 - synapse.http.client - 924 - INFO - PUT-5624 - Received response to POST synapse-replication://matrix-synapse-worker-stream-writer-3-account-data/_synapse/replication/add_tag/%40test7%3Aourmatrixdomain.net/%21dEMKYIHEGChqUWMutz%3Aourmatrixdomain.net/m.server_notice: 200
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,043 - synapse.server_notices.server_notices_manager - 199 - INFO - PUT-5624 - Created server notices room !dEMKYIHEGChqUWMutz:ourmatrixdomain.net for @test7:ourmatrixdomain.net
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,047 - synapse.http.server - 114 - INFO - PUT-5624 - SynapseError: 429 - Too Many Requests
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,049 - synapse.access.http.8008 - 461 - INFO - PUT-5624 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.277sec/0.001sec (0.027sec, 0.008sec) (0.013sec/0.064sec/50) 80B 429 "PUT /_synapse/admin/v1/send_server_notice/@test7:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [2 dbevts]
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,676 - synapse.server_notices.server_notices_manager - 152 - INFO - PUT-5634 - Using existing server notices room !kNOowWmoRFgmmigJQX:ourmatrixdomain.net for user @test:ourmatrixdomain.net
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,684 - synapse.server_notices.server_notices_manager - 76 - INFO - PUT-5634 - Sending server notice to @test:ourmatrixdomain.net
Jul 19 12:34:53 mxbhv-app-mas-nbg matrix-synapse[3355891]: 2023-07-19 12:34:53,749 - synapse.access.http.8008 - 461 - INFO - PUT-5634 - ::ffff:172.18.0.24 - 8008 - {@root:ourmatrixdomain.net} Processed request: 0.084sec/0.001sec (0.012sec, 0.002sec) (0.006sec/0.027sec/25) 59B 200 "PUT /_synapse/admin/v1/send_server_notice/@test:ourmatrixdomain.net.ns64b7d814ada6f8.93327386 HTTP/1.0" "Symfony HttpClient/Curl" [3 dbevts]
```
### Anything else that would be useful to know?
So looking at the logs, messages go through when the server notices room already exists. If it has to be created, the request is rate limited, but it looks like the room is still created. I don't get that logic.
Contributor guide
Assessment
This issue has not been assessed yet.