apache / apache/trafficserver

Reverse DNS (PTR) lookup for raw IP origins causes ~5s latency on every request including cache hits

Open
#12,980 0 comments 0 reactions 0 assignees View on GitHub
Bug Cache DNS HostDB HTTP Parent Proxy
Dominant language
C++
Stars
2k
Forks
874
Avg merge
6d 15h
Merged PRs (30d)
46

Description

## Problem

When `remap.config` maps to a raw IP origin (e.g., `map http://cdn.example.com/ http://203.x.x.x/`) and `parent.config` has any `dest_domain=` entries (even for unrelated services), ATS performs a PTR (reverse DNS) lookup on **every request**, including cache HIT-FRESH.

If the upstream DNS returns ServFail for these PTR queries, ATS retries up to `proxy.config.dns.retries` times (default 5), adding ~5 seconds to every response.

### Root Cause

Two issues compound to create this behavior:

**1. `hostMatch` forces DNS (including reverse DNS) before cache lookup**

In `HandleRequest` ([src/proxy/http/HttpTransact.cc:1636-1642](https://github.com/apache/trafficserver/blob/master/src/proxy/http/HttpTransact.cc#L1636-L1642)), when `parent.config` has any `dest_domain`/`dest_host` rules (`hostMatch` is true), `force_dns` is set for all requests -- including those with raw IP origins:

```cpp
if (s->server_info.name_addr.is_valid() &&
(!s->state_machine->enable_redirection || !s->redirect_info.redirect_in_process) &&
s->parent_params->parent_table->hostMatch) {
s->force_dns = true;
}
```

Then in `OSDNSLookup` ([src/proxy/http/HttpTransact.cc:2024-2028](https://github.com/apache/trafficserver/blob/master/src/proxy/http/HttpTransact.cc#L2024-L2028)), the reverse DNS is dispatched:

```cpp
} else if (s->server_info.name_addr.is_valid() && s->parent_params->parent_table->hostMatch &&
!s->txn_conf->no_dns_forward_to_parent) {
// note, broken logic: ACC fudges the OR stmt to always be true,
// 'AuthHttpAdapter' should do the rev-dns if needed, not here .
TRANSACT_RETURN(StateMachineAction_t::DNS_REVERSE_LOOKUP, HttpTransact::StartAccessControl);
```

Note the existing source comment acknowledging this as "broken logic."

**2. Negative DNS results are not cached by default**

`proxy.config.hostdb.fail.timeout` defaults to `0`, which means failed DNS lookups (ServFail, NXDOMAIN) are never cached in HostDB. Every request triggers a fresh PTR lookup with full retries, rather than caching the failure and short-circuiting subsequent requests.

### Request Flow (current)

```
Request -> HandleRequest -> force_dns=true (hostMatch) -> Forward DNS (instant for IP)
-> OSDNSLookup -> Reverse DNS/PTR (ServFail x 5 retries = ~5s)
-> StartAccessControl -> Cache Lookup -> Cache HIT -> Serve (after ~5s delay)
```

### Key Observations

- **Cache hits never need parent selection.** The PTR result feeds `hostname_str`, which is used for `dest_domain`/`dest_host` matching in `parent.config`. Parent selection only matters on cache misses.
- **The cache key is URL-based, not DNS-based.** Cache keys are constructed from the request URL host (via `url->host_get()`), not from `hostname_str`. The PTR result never influences cache lookup.
- **`hostMatch` is a global flag.** If *any* `dest_domain` rule exists in `parent.config`, *all* requests get forced through the reverse DNS path, even origins that would match via `dest_ip` rules.
- **`parent.config` ipMatch also forces DNS before cache unnecessarily.** `parent_table->ipMatch` is one of the conditions for `force_dns` in [HttpSM.cc:347-349](https://github.com/apache/trafficserver/blob/master/src/proxy/http/HttpSM.cc#L347-L349), but parent selection is only needed on cache miss.

### Existing Workaround

Setting `proxy.config.cache.hostdb.disable_reverse_lookup INT 1` eliminates the PTR delay. HostDB returns nullptr immediately for reverse lookups. Side effect: `hostname_str` stays as the raw IP string, so `dest_domain`/`dest_host` parent rules will not match against PTR-resolved hostnames (but `dest_ip` rules are unaffected).

## Proposed Options

### Option 1: Defer parent.config DNS to after cache lookup (Recommended)

Remove `parent.config` triggers from `force_dns` and move reverse DNS to the cache-miss path.

**Changes:**
- Remove `parent_table->ipMatch` from the `force_dns` calculation in `HttpSM.cc:347-349`
- Remove the `hostMatch` -> `force_dns` block in `HttpTransact.cc:1636-1642`
- Remove the `DNS_REVERSE_LOOKUP` dispatch in `OSDNSLookup` (`HttpTransact.cc:2024-2028`)
- Add forward DNS (for `ipMatch`) and reverse DNS (for `hostMatch`) to the cache-miss path, before parent selection

**Pros:**
- Cache hits skip all parent.config-related DNS (forward and reverse)
- `cache.config` `dest_ip` rules still work (that trigger is independent and legitimate)
- `doc_in_cache_skip_dns` and `TS_HTTP_OS_DNS_HOOK` behavior unchanged
- Clean separation: cache.config gates cache decisions (needs DNS first), parent.config gates origin routing (only needs DNS on miss)

**Cons:**
- Touches core state machine flow
- Need to verify no plugin depends on `hostname_str` being set from PTR before cache lookup

### Option 2: Change `hostdb.fail.timeout` default from 0 to 30

Cache failed DNS results for 30 seconds so subsequent requests do not re-trigger the full retry cycle.

**Changes:**
- Change the default for `proxy.config.hostdb.fail.timeout` from `0` to `30` in `RecordsConfig.cc`

**Pros:**
- Simple one-line change
- Helps all DNS failure scenarios, not just PTR
- Aligns with RFC 2308 (negative caching)
- First request still pays the penalty, but subsequent requests within 30s are instant

**Cons:**
- Does not fix the architectural issue (reverse DNS still runs before cache for every request)
- A host that recovers will not be retried for up to 30 seconds
- First request to a new ServFail PTR still takes ~5 seconds

### Option 3: Option 1 + Option 2 combined (Recommended)

Defer parent.config DNS to after cache lookup AND enable negative caching by default.

**Pros:**
- Cache hits: zero DNS overhead (no forward, no reverse)
- Cache misses with failing DNS: 30s negative cache window prevents retry storms
- Addresses both the architectural issue and the operational pain

**Cons:**
- More changes to review and test

### Option 4: Skip reverse DNS for IP-literal origins entirely

When the origin in remap.config is already an IP address (not a hostname), never dispatch `DNS_REVERSE_LOOKUP` for it. The PTR result for a raw IP origin is almost never useful -- if the operator wanted hostname-based parent matching, they would have used a hostname in remap.config.

**Changes:**
- In `OSDNSLookup`, before dispatching `DNS_REVERSE_LOOKUP`, check if the origin is an IP literal and skip the PTR

**Pros:**
- Targeted fix for the exact scenario reported
- Minimal code change
- Does not change behavior for hostname origins

**Cons:**
- Does not address the broader issue of unnecessary DNS before cache lookup
- Edge case: an operator could legitimately want PTR-based parent matching for IP origins (unlikely)

## Recommendation

**Option 3 (Option 1 + Option 2)** provides the most comprehensive fix:

- Option 1 fixes the architecture: parent.config DNS is deferred to when it is actually needed (cache miss). This is the right long-term fix -- the source code already acknowledges the current approach as "broken logic."
- Option 2 is independently valuable: negative DNS caching should be enabled by default. A default of 0 (never cache failures) means any DNS failure triggers full retries on every request, which is a problem beyond just PTR lookups.

For users on older versions (9.x), the workaround `proxy.config.cache.hostdb.disable_reverse_lookup INT 1` is safe when parent matching for raw IP origins uses `dest_ip` rules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.