Calls not connecting with Wireguard VPN

**Describe the bug**

I have a VPN server at home to connect to my homelab. I connect to it using wireguard on android. The VPN configuration only allow the subnet `192.168.255.0/24` to the tunnel.

When my vpn is up on the phone, calls no longer connect. I see "Call connecting..." forever. With the VPN down, everything works fine.

**To Reproduce**
Steps to reproduce the behavior:
1. Start the vpn on the phone
2. Make a call to someone (1-to-1 call)
3. Call stuck in "Call connecting..." state

**Expected behavior**

I would expect the call to connect since only traffic within `192.168.255.0/24` goes through the VPN.

**Smartphone (please complete the following information):**
- Device: Xiaomi, Mi A2
- OS: Android 8.1.0

**Additional context**

- App version and store: 1.0.14 - F-Droid
- Homeserver: matrix.org

I see a similar behavior in Jitsi, calls connect but I don't get any audio/video from the other person, and the person doesn't get any audio/video from me.

Here's the configuration file for wireguard for the phone:

```ini
[Interface]
Address = 192.168.255.2/24
PrivateKey =

[Peer]
AllowedIPs = 192.168.255.0/24
Endpoint = 192.168.0.175:51820
PublicKey = +883cYmWGvYEkjZW7JBinmxx/zqhx+LnwFI3MOWPjlY=
```