element-hq / element-hq/synapse
Pusher stops pushing when it falls behind, with lots of DNSLookupFailed errors
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#7113](https://github.com/matrix-org/synapse/issues/7113).
---
Our pusher process got behind. A couple of hours after it started to catch up, almost all pushes started failing:

Inspection of logs showed that they were failing with DNSLookupFailed errors:
```
2020-03-19 11:11:27,083 - synapse.http.client - 281 - INFO - httppush.process-57923 - Sending request POST http://10.103.0.7/_matrix/push/v1/notify
2020-03-19 11:12:28,647 - synapse.http.client - 328 - INFO - httppush.process-57923 - Error sending request to POST http://10.103.0.7/_matrix/push/v1/notify: DNSLookupError Couldn't find the hostname '10.103.0.7'
2020-03-19 11:12:28,647 - synapse.push.httppusher - 391 - WARNING - httppush.process-57923 - Failed to push event to : DNS lookup failed: Couldn't find the hostname '10.103.0.7'.
2020-03-19 11:12:32,225 - synapse.push.httppusher - 276 - INFO - httppush.process-57923 - Push failed: delaying for 1s
```
Contributor guide
Assessment
This issue has not been assessed yet.