stubby: Stubby spams log with “Could not schedule query” when internet goes down

Maintainer: @jonathanunderwood
Environment: BT Home Hub 5A, xRX200 rev 1.2, OpenWrt 19.07.4

Description:

When the WAN goes down (which I have been unlucky to have happen a lot recently due to local problems), stubby fills the log with this message (about 110 a second):

Tue Oct  6 12:22:41 2020 daemon.err stubby[4098]: Could not schedule query: None of the configured upstreams could be used to send queries on the specified transports

I have (I believe) a standard stubby configuration, having followed the guide:

stubby.global=stubby
stubby.global.manual='0'
stubby.global.trigger='wan'
stubby.global.dns_transport='GETDNS_TRANSPORT_TLS'
stubby.global.tls_authentication='1'
stubby.global.tls_query_padding_blocksize='128'
stubby.global.appdata_dir='/var/lib/stubby'
stubby.global.edns_client_subnet_private='1'
stubby.global.idle_timeout='10000'
stubby.global.round_robin_upstreams='1'
stubby.global.tls_connection_retries='2'
stubby.global.listen_address='127.0.0.1@5453'
stubby.@resolver[0]=resolver
stubby.@resolver[0].address='1.1.1.1'
stubby.@resolver[0].tls_auth_name='cloudflare-dns.com'
stubby.@resolver[1]=resolver
stubby.@resolver[1].address='1.0.0.1'
stubby.@resolver[1].tls_auth_name='cloudflare-dns.com'
stubby.@resolver[2]=resolver
stubby.@resolver[2].address='8.8.4.4'
stubby.@resolver[2].tls_auth_name='dns.google'
stubby.@resolver[3]=resolver
stubby.@resolver[3].address='8.8.8.8'
stubby.@resolver[3].tls_auth_name='dns.google'

I would have hoped that there would be some sort of backoff after a failure. As it is, it is quite difficult to see what has happened as the logs get completely filled with these messages.

Start by tracing the stubby log path that emits “Could not schedule query” and reproduce it while the WAN is unavailable, using the configuration and OpenWrt environment described here. Done means repeated upstream scheduling failures are backed off or otherwise rate-limited, while DNS queries recover when connectivity returns; add or run a regression test if the project has coverage for this path.