networkd, nspawn, resolved, and slow DNS replies

systemd version the issue has been seen with
systemd 238

Used distribution
Fedora 28

Unexpected behavior you saw
Nspawn containers have network connectivity issues.
After the container is up and running, there are difficulties to connect from the containers to the outside world. A simple ping to some domain address outside the network takes sometimes 3-4 seconds to resolve to an IP address, and another 10-20 seconds to return with the first ping - where the ping answer is just a few milliseconds.

Steps to reproduce the problem

Create containers with the following config

[Unit]
Description=srvctl - container %i
Documentation=man:systemd-nspawn(1)
PartOf=machines.target
Before=machines.target
After=network.target

[Service]
ExecStartPre=/bin/bash /usr/local/share/srvctl/modules/containers/execstartpre.sh %i
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --settings=trusted --machine=%i -D /srv/%i/rootfs
ExecStartPost=/bin/bash /usr/local/share/srvctl/modules/containers/execstartpost.sh %i
ExecStopPost=/bin/bash /usr/local/share/srvctl/modules/containers/execstoppost.sh %i

KillMode=mixed
Type=notify
RestartForceExitStatus=133
SuccessExitStatus=133
Slice=machine.slice
Delegate=yes
TasksMax=16384

# Enforce a strict device policy, similar to the one nspawn configures
# when it allocates its own scope unit. Make sure to keep these
# policies in sync if you change them!
DevicePolicy=closed
DeviceAllow=/dev/net/tun rwm
DeviceAllow=char-pts rw

# nspawn itself needs access to /dev/loop-control and /dev/loop, to
# implement the --image= option. Add these here, too.
DeviceAllow=/dev/loop-control rw
DeviceAllow=block-loop rw
DeviceAllow=block-blkext rw

[Install]
WantedBy=machines.target

The host and the containers both run systemd-networkd, and systemd-resolved where /etc/resolve.conf points to the host interface with a single nameserver entry. The host pings everything fine.

[root@test ~]# networkctl status
●        State: routable
       Address: 10.20.0.15 on host0
                fe80::68a8:b3ff:feba:bd09 on host0
       Gateway: 10.20.0.1 on host0

After the domain has been pinged once, connectivity is OK. - Sometimes.

[root@pmtest ~]# time ping freedesktop.org
PING freedesktop.org (131.252.210.176) 56(84) bytes of data.
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=1 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=2 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=3 ttl=47 time=163 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=4 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=5 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=6 ttl=47 time=162 ms
^C
--- freedesktop.org ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 27192ms
rtt min/avg/max/mdev = 162.159/162.762/163.018/0.596 ms

real	0m41.980s
user	0m0.002s
sys	0m0.005s
[root@pmtest ~]# time ping freedesktop.org
PING freedesktop.org (131.252.210.176) 56(84) bytes of data.
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=1 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=2 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=3 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=4 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=5 ttl=47 time=162 ms
64 bytes from 131.252.210.176 (131.252.210.176): icmp_seq=6 ttl=47 time=162 ms
^C
--- freedesktop.org ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 27188ms
rtt min/avg/max/mdev = 162.025/162.359/162.910/0.479 ms

real	0m28.061s
user	0m0.002s
sys	0m0.004s
[root@pmtest ~]# time ping freedesktop.org
PING freedesktop.org (131.252.210.176) 56(84) bytes of data.
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=1 ttl=47 time=162 ms
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=2 ttl=47 time=162 ms
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=3 ttl=47 time=162 ms
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=4 ttl=47 time=162 ms
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=5 ttl=47 time=162 ms
64 bytes from annarchy.freedesktop.org (131.252.210.176): icmp_seq=6 ttl=47 time=163 ms
^C
--- freedesktop.org ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 162.111/162.656/163.077/0.337 ms

real	0m5.538s
user	0m0.001s
sys	0m0.004s

What could I do about this?

Start by reviewing the supplied systemd-nspawn unit and reproducing the delay with systemd-networkd and systemd-resolved inside a Fedora 28 container. Compare the container's DNS resolution and ping timing with the host, then use the reported behavior to determine what needs correction; done means external name resolution and the first ping no longer incur the multi-second delays.