element-hq / element-hq/synapse
ResponseNeverReceived isn't always a timeout
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#9662](https://github.com/matrix-org/synapse/issues/9662).
---
### Description
A Synapse installation configured with OIDC started failing after upgrading 1.21 -> 1.26/1.28 with `RequestTimedOutError: 504: Timeout waiting for response from remote server`. The installation is setup with a HTTP/HTTPS CONNECT proxy.
On investigation it turned out that Synapse was unable to verify the TLS certificate of the OIDC IdP with the default CA trust store.
I've changed the trust store by explicitly setting the `SSL_CERT_FILE` environment variable but this issue made it so the customer couldn't see the actual problem and took me editing the installed Python to discover it.
### Steps to reproduce
- Synapse running in an environment that requires a HTTP/HTTPS CONNECT proxy
- Synapse configured with OIDC
- The default trust store (somewhat unsure which this is in this environment) can't verify the chain
```
....
Traceback (most recent call last):
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 97, in load_metadata
await p.load_metadata()
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 372, in load_metadata
metadata_response = await self._http_client.get_json(url)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 570, in get_json
body = await self.get_raw(uri, args, headers=headers)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 655, in get_raw
response = await self.request("GET", uri, headers=Headers(actual_headers))
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 422, in request
response = await make_deferred_yieldable(request_deferred)
File "/opt/synapse/env/lib64/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 750, in _timeout_to_request_timed_out_error
raise RequestTimedOutError("Timeout waiting for response from remote server")
synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/_base.py", line 159, in wrapper
await cb(*args, **kwargs)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/homeserver.py", line 431, in start
await oidc.load_metadata()
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 102, in load_metadata
) from e
Exception: Error while initialising OIDC provider 'oidc'
```
Should have been:
```
...
Traceback (most recent call last):
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 97, in load_metadata
await p.load_metadata()
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 372, in load_metadata
metadata_response = await self._http_client.get_json(url)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 570, in get_json
body = await self.get_raw(uri, args, headers=headers)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 655, in get_raw
response = await self.request("GET", uri, headers=Headers(actual_headers))
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/http/client.py", line 422, in request
response = await make_deferred_yieldable(request_deferred)
twisted.web._newclient.ResponseNeverReceived: []
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/_base.py", line 159, in wrapper
await cb(*args, **kwargs)
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/app/homeserver.py", line 431, in start
await oidc.load_metadata()
File "/opt/synapse/env/lib64/python3.6/site-packages/synapse/handlers/oidc_handler.py", line 102, in load_metadata
) from e
Exception: Error while initialising OIDC provider 'oidc'
```
### Version information
- **Version**: Experienced with 1.26 / 1.28. Was not happening on 1.21
- **Install method**:
- Custom RPM built for S390x on RHEL7
- The RPM is effectively a virtual environment with `rh-python36`
Contributor guide
Assessment
This issue has not been assessed yet.