Ratelimit RPC + Cache configuration of DNS servers & prepared queries issue

I would like to report a bug that cause a small outage on our side and dicuss how it could be fixed.

We are running DNS Consul service on very few DNS machines that are behing powerdns machines and redirect our Consul zone on those consul agents.

# Context

We are running 1.8.4 machines with the following DNS configurations:

```json
"dns_config": {
"allow_stale": true,
"only_passing": true,
"node_ttl": "1m",
"use_cache": true,
"cache_max_age": "5s",
"service_ttl": {
"*": "10s",
"some-service-prefix*": "1s"
},
"soa": {
"expire": 86400,
"min_ttl": 30,
"refresh": 3600,
"retry": 600
},
"a_record_limit": 1,
"enable_additional_node_meta_txt": false
},
```

=> Note the cache_max_age: 5s

A few weeks ago, we introduced cache-per-entry rate limiting on all our agents, first softly with rate limit of 1 query every 5s, with:
```json
{
"cache": {
"entry_fetch_rate": 0.2,
"entry_fetch_max_burst": 4
}
}
```

and everything was working well.

We also use prepared queries similar to that:
```json
{
"Template": {
"Type": "name_prefix_match"
},
"Service": {
"Service": "${name.full}",
"Failover": {
"NearestN": 3
},
"OnlyPassing": false
}
}
```

=> do doing a redis.query.consul DNS lookup will find instances of redis in local DC if any, in any of 3 closest ones otherwise.

=> so rate limit of 1 RPC qps of a given cache entry (for instance, for queries on serviceX -> only 1 update every 5s), with a burst of 4 calls.

# The issue

But when we switched to more aggressive ratelimit (1 RPC every 10s):
```json
{
"cache": {
"entry_fetch_rate": 0.1,
"entry_fetch_max_burst": 4
}
}
```

Things started to break for prepared queries on our DNS servers: our DNS server did spend a lot of time answering queries and created lots of timeouts.

The reason is that:
* `"cache_max_age": "5s"` => will flush cache every 5s
* `"entry_fetch_rate": 0.1`=> will allow a request every 10s max.

Thus, for DNS prepared queries, since requests are not in flight all the time (such as for service watches), the cache is emptied every 5s, and when the next DNS query is sent, it has to wait for the rate limiter.

As long as the `cache_max_age`is lower than the time between each cache requests, everything is fine, but when threshold is reached, it breaks, becase DNS queries wait for the rate-limiter.

# Local reproduction (without any kind of cluster)

## Launch an agent (working)
```shell
consul agent -dev -hcl 'dns_config { allow_stale = true, only_passing = true, cache_max_age = "6s", use_cache = true }' -hcl 'cache { entry_fetch_rate = 0.2}'
```

## Register a prepared query
```shell
curl --request POST --data '{ "Template": { "Type": "name_prefix_match" }, "Service": { "Service": "${name.full}", "Failover": { "NearestN": 3 }, "OnlyPassing": false }}' localhost:8500/v1/query
```

## Launch a loop of dns queries
```
while time dig @localhost -p 8600 consul.query.consul +short; do printf .; done
```

This initial test should be working.

But if you use `dns_config.cache_max_age` < `cache.ratelimit per entry`, it starts slowing down requests, and you will end with something like (here with `dns_config.cache_max_age = 1s` and `cache.entry_fetch_rate = 0.2`:

```
.127.0.0.1
4.99 real 0.00 user 0.00 sys
.127.0.0.1
4.99 real 0.00 user 0.00 sys
.127.0.0.1
4.99 real 0.00 user 0.00 sys
.127.0.0.1
4.99 real 0.00 user 0.00 sys
```

So, 5s to answer any DNS query.

Which is actually expected (there is a rate limit of 1 query every 5s for prepared queries), but unlikely the result wanted with DNS (aka very slow requests).

What is really fun here, is that it happens only with prepared queries, not with dig consul.service.consul -> because a watch is started and ready to give the result while prepared queries are not using watches.

# Possible fix strategies

## 1. Ensure that parameters are consistent and delay implied by `cache.entry_fetch_rate`is lower than `dns_config.cache_max_age`

If would be possible at startup of consistency of parameters:
```
if dns_config.use_cache && cache.entry_fetch_rate != +Inf {
rate_limiter_interval = 1s / cache.entry_fetch_rate
if dns_config.cache_max_age < rate_limiter_interval {
// FAILED CONFIG FILE validation
refuse_to_start()
}
}
```

## 2. Ensure that cache for DNS queries is not emptied by `dns_config.cache_max_age`

Might be abit more complicated, but more clever...

## 3. Ignore rate-limiter, when cache-entry is empty

If a cache entry is empty and discarded, never apply the rate limiter.

## 4. Handle cache of prepared queries differently

Did not look on how doing that yet.

# Advice from maintainers

On our side, in the meantime, we will ensure at infrastructure level that `cache.entry_fetch_rate`is lower than `dns_config.cache_max_age`.

This is a basic analysis (did not look closely at the implementation), but I think it might desserve a strategy to avoid people having the same issue (I agreee, it is not trivial).

Fix strategy 1. is easy to implement, but maybe you think that's not enough.

Do you have any specific ideas regarding this issue?

Reproduce the behavior with the consul agent, prepared-query registration, and dig loop described in the issue, comparing cache_max_age with entry_fetch_rate. Then trace the DNS prepared-query path, cache expiration, and per-entry rate limiter; done means prepared-query DNS requests no longer incur repeated rate-limit delays when the cache expires.

Written by the indexing model from the issue text.