element-hq / element-hq/dendrite
federation: Server without IPv6 support retries failed IPv6 connections without back off, spamming .well-known lookups many times a second
- Dominant language
- Go
- Stars
- 965
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
*This issue was originally created by [**@Ablu**](https://github.com/Ablu) at .*
### Background information
- **Dendrite version or git SHA**: `0.13.5+b7054f4`
- **SQLite3 or Postgres?**: `unknown`
- **Running in Docker?**: yes
- **`go version`**: `unknown`
- **Client used (if applicable)**: `na`
### Description
- **What** is the problem:
If a server does not support IPv6, but another server is only reachable via IPv6 connections are attempted in a tight loop without any back-off. This results in many .well-known lookups on the target server per second.
- **Who** is affected: The IPv6-only server is getting hammered by .well-known request non-stop. The IPv4-only server gets a huge amount of log errors
- **How** is this bug manifesting: Spam in logs, rate-limiting kicking in and DDoSing the target server.
- **When** did this first appear: unknown, noticed it when I got rate limited by my webhosting provider that hosted the .well-known file.
### Steps to reproduce
- Run dendrite on a IPv4 only host
- Attempt federation with an IPv6 only host (For example, message `@ablu:ablu.org`)
- Observe non-stop log errors:
```
time="2023-12-18T15:14:28.942253091Z" level=debug msg="Error sending request to https://matrix.ablu.org:443/_matrix/key/v2/server: dial tcp [2001:9e8:d58b:8100:51be:e7e4:9b58:7507]:443: connect: network is unreachable" out.req.ID=lpqzvlnCpkM4 out.req.method=GET out.req.uri="matrix://ablu.org/_matrix/key/v2/server" req.id=1tjUd9EkbTnE req.method=PUT req.path=/_matrix/federation/v1/send/1699241410034
time="2023-12-18T15:14:28.949413244Z" level=debug msg="Error sending request to https://matrix.ablu.org:443/_matrix/key/v2/server: dial tcp [2001:9e8:d58b:8100:51be:e7e4:9b58:7507]:443: connect: network is unreachable" out.req.ID=lpqzvlnCpkM4 out.req.method=GET out.req.uri="matrix://ablu.org/_matrix/key/v2/server" req.id=1tjUd9EkbTnE req.method=PUT req.path=/_matrix/federation/v1/send/1699241410034
time="2023-12-18T15:14:28.949483724Z" level=debug msg="Outgoing request failed" error="Get \"matrix://ablu.org/_matrix/key/v2/server\": dial tcp [2001:9e8:d58b:8100:51be:e7e4:9b58:7507]:443: connect: network is unreachable" out.req.ID=lpqzvlnCpkM4 out.req.method=GET out.req.uri="matrix://ablu.org/_matrix/key/v2/server" req.id=1tjUd9EkbTnE req.method=PUT req.path=/_matrix/federation/v1/send/1699241410034
time="2023-12-18T15:14:28.979877904Z" level=debug msg="Error sending request to https://matrix.ablu.org:443/_matrix/key/v2/query: dial tcp [2001:9e8:d58b:8100:51be:e7e4:9b58:7507]:443: connect: network is unreachable" out.req.ID=av7ZmKa6ZhFR out.req.method=POST out.req.uri="matrix://ablu.org/_matrix/key/v2/query" req.id=1tjUd9EkbTnE req.method=PUT req.path=/_matrix/federation/v1/send/1699241410034
time="2023-12-18T15:14:28.981335511Z" level=debug msg="Error sending request to https://matrix.ablu.org:443/_matrix/key/v2/query: dial tcp [2001:9e8:d58b:8100:51be:e7e4:9b58:7507]:443: connect: network is unreachable" out.req.ID=av7ZmKa6ZhFR out.req.method=POST out.req.uri="matrix://ablu.org/_matrix/key/v2/query" req.id=1tjUd9EkbTnE req.method=PUT req.path=/_matrix/federation/v1/send/1699241410034
```
- Observe matching .well-known lookups on the target server
# Expected behaviour:
The connection should fail and exponentially back off. The .well-known entry should probably be cached as well.
/cc @davralin
Contributor guide
Research direction
The issue names Dendrite federation requests and .well-known lookups but no source files or tests. Start by tracing the federation request path while reproducing an IPv4-only host contacting an IPv6-only host. Done means failed connections back off exponentially and repeated .well-known lookups no longer hammer the target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100