element-hq / element-hq/synapse

Pusher stops pushing when it falls behind, with lots of DNSLookupFailed errors

Open
#7,113 0 comments 1 reaction 0 assignees View on GitHub
A-Performance A-Push T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#7113](https://github.com/matrix-org/synapse/issues/7113).

---

Our pusher process got behind. A couple of hours after it started to catch up, almost all pushes started failing:

![image](https://user-images.githubusercontent.com/1389908/77156873-31206000-6a98-11ea-8855-6102545355fa.png)

Inspection of logs showed that they were failing with DNSLookupFailed errors:

```
2020-03-19 11:11:27,083 - synapse.http.client - 281 - INFO - httppush.process-57923 - Sending request POST http://10.103.0.7/_matrix/push/v1/notify
2020-03-19 11:12:28,647 - synapse.http.client - 328 - INFO - httppush.process-57923 - Error sending request to POST http://10.103.0.7/_matrix/push/v1/notify: DNSLookupError Couldn't find the hostname '10.103.0.7'
2020-03-19 11:12:28,647 - synapse.push.httppusher - 391 - WARNING - httppush.process-57923 - Failed to push event to : DNS lookup failed: Couldn't find the hostname '10.103.0.7'.
2020-03-19 11:12:32,225 - synapse.push.httppusher - 276 - INFO - httppush.process-57923 - Push failed: delaying for 1s
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.