element-hq / element-hq/synapse

Deferred Cancellation appears to make network errors more obscure (e.g. `ConnectingCancelledError: HostnameAddress(...)`)

Open
#12,706 0 comments 0 reactions 0 assignees View on GitHub
T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#12706](https://github.com/matrix-org/synapse/issues/12706).

---

I was running Complement with the `ufw` firewall enabled and this seems to have blocked outgoing connections from the container to the host (`host.docker.internal`).

The logs (with a [not very useful] patch from `logger.warning` to `logger.exception`) obtained were:
```
2022-05-11 11:17:39,874 - synapse.crypto.keyring - 825 - ERROR - ServerKeyFetcher-0 - Error looking up keys ['ed25519:complement'] from host.docker.internal:33145: Failed to send request: ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145)
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/http/matrixfederationclient.py", line 582, in _send_request
response = await make_deferred_yieldable(request_deferred)
twisted.internet.error.ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/crypto/keyring.py", line 858, in get_server_verify_key_v2_direct
response = await self.client.get_json(
File "/usr/local/lib/python3.9/site-packages/synapse/http/matrixfederationclient.py", line 1062, in get_json
response = await self._send_request_with_optional_trailing_slash(
File "/usr/local/lib/python3.9/site-packages/synapse/http/matrixfederationclient.py", line 389, in _send_request_with_optional_trailing_slash
response = await self._send_request(request, **send_request_args)
File "/usr/local/lib/python3.9/site-packages/synapse/http/matrixfederationclient.py", line 586, in _send_request
raise RequestSendFailed(e, can_retry=True) from e
synapse.api.errors.RequestSendFailed: Failed to send request: ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/crypto/keyring.py", line 822, in get_key
keys = await self.get_server_verify_key_v2_direct(server_name, key_ids)
File "/usr/local/lib/python3.9/site-packages/synapse/crypto/keyring.py", line 879, in get_server_verify_key_v2_direct
raise KeyLookupError(str(e))
synapse.crypto.keyring.KeyLookupError: Failed to send request: ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145)
2022-05-11 11:17:39,877 - synapse.federation.transport.server._base - 306 - WARNING - GET-4 - authenticate_request failed: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='host.docker.internal:33145', minimum_valid_until_ts=1652267849856, key_ids=['ed25519:complement'])
2022-05-11 11:17:39,878 - synapse.http.server - 165 - INFO - GET-4 - SynapseError: 401 - Failed to find any key to satisfy: _FetchKeyRequest(server_name='host.docker.internal:33145', minimum_valid_until_ts=1652267849856, key_ids=['ed25519:complement'])
```

`ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145)` isn't a really useful error message. If anything, it reminded me of some nonsense error messages that I've seen from Twisted when an untrusted TLS certificate (or other alert condition) is encountered — if I recall that has similar wording around 'cancellation'. Alternatively it makes it sound like a DNS lookup problem.

I'm not sure how we can easily make this better (perhaps by wrapping the error if a request is cancelled, giving a clearer message), but I found this quite pesky to see what was going on.

As a note, it was `timeout_deferred` → `time_it_out` that called `deferred.cancel()` to lead to this particular case.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.