Consul Connect/xDS on client node enters persistent ACL not found state while workload proxy token remains valid

Linked from https://github.com/hashicorp/nomad/issues/28209
Initial inspection suggests this is a Consul bug and requires investigation from the team here.

Nomad v1.11.3
BuildDate 2026-03-11T09:10:27Z
Revision 173ab08a0210789da531847c4ce3c3518f7fb34b

Consul v1.22.2
Revision 71f21bf3
Build Date 2025-12-17T05:40:06Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

### Operating system and Environment details
- OS: RHEL 8.10
- Nomad workloads with Connect sidecars
- Consul workload identities via auth method `nomad-workloads`
- affected node in the example below: `q10i22`
- renewal of Consul + Nomad Certs and Tokens via vault-agent

### Issue

We are seeing a Consul Connect/xDS failure mode on a Nomad client node where the local Consul client starts returning `ACL not found` for many Connect-related RPCs such as:

- `ConfigEntry.Get`
- `ConfigEntry.List`
- `ConfigEntry.ResolveServiceConfig`
- `Intention.Match`
- `DiscoveryChain.Get`
- `ConnectCA.Sign`

Once the node enters this state, affected sidecar proxies on that node repeatedly log:

- `DeltaAggregatedResources gRPC config stream to local_agent closed: 10, xDS stream terminated due to an irrecoverable error`

The important part is: the affected workload proxy token can still be valid and readable via `consul acl token read -self`.

So this does not currently look like a simple case of an expired or deleted workload token.

### Example affected workload

- allocation: `12f978f7-bd96-f83d-339d-8c8fa2f6a97f`
- task group: `worker-long`
- proxy task: `connect-proxy-sdsapi-worker`
- proxy token accessor: `ea36de02-148c-efe5-43c0-cfee1312ad29`

The corresponding token is still valid at collection time:

- `Description: token created via login: {"requested_by":"nomad_service_sdsapi-worker"}`
- `AuthMethod: nomad-workloads`
- `ExpirationTime: null`

### What we observe

#### Node-level Consul failure

On `q10i22`, the local Consul client already shows persistent Connect/ACL failures at the beginning of the collection window:

- `2026-07-02 19:00:01 CEST` -> `ConfigEntry.Get -> ACL not found`
- `2026-07-02 19:00:05 CEST` -> `ConfigEntry.List -> ACL not found`
- `2026-07-02 19:00:10 CEST` -> `Intention.Match -> ACL not found`
- `2026-07-02 19:00:17 CEST` -> `ConnectCA.Sign -> ACL not found`
- `2026-07-02 19:00:17 CEST` -> `agent.leaf-certs -> ACL not found`

#### Two-phase behavior

We see two phases:

1. Low-rate background phase
- from about `19:00` to `23:32`
- roughly `1730-1748` `ACL not found` lines per hour
- usually around `16-44` per minute

2. Escalation phase
- starts around `2026-07-02 23:33 CEST`
- error rate jumps sharply
- hourly counts then rise to:
- `2026-07-02 23`: `27316`
- `2026-07-03 00`: `59285`

So the node appears to be already unhealthy before the visible error storm begins (but we see no affect on our applications).

#### Proxy-side effect

The affected proxy then repeatedly logs:

- `xDS stream terminated due to an irrecoverable error`
- interleaved with:
- `cds: response indicates 0 added/updated cluster(s), 0 removed cluster(s)`

This suggests:

- the proxy is still running
- it can still talk to the local agent socket/path
- but the local agent cannot provide stable usable xDS results

We are **not** claiming that every Connect proxy on the node shows visible xDS errors at exactly the same time.
We do have an example on the same node (`0ba62444-7268-d158-0304-230b81f3ae62`, `connect-proxy-vmetrics-pfit`) that did not show visible xDS errors at the same time, while the node-level Consul error state was still present.

So the strongest claim we can currently support is:
- the Consul client node enters a broken Connect/ACL state
- some workloads on that node are then visibly affected
- the blast radius may be partial or time-skewed

## Why this seems like a bug

The affected workload token is still valid and readable, but the local Consul client continues to fail host-locally with `ACL not found` across many Connect-related RPCs.

This looks more like:

- broken or stale ACL/cache state in the local client
- or a bad interaction in Connect/xDS/ACL handling on the client node

than a straightforward “the workload token expired”.

## Workaround / operational impact

- restarting only the app container is not sufficient
- restarting only the proxy container is not sufficient
- restarting the full allocation restores connectivity temporarily
- the broken node state persist for hours until manual intervention

### Reproduction steps

Unfortunately ... starting a Nomad Application with consul connect proxy connection and wait :-/

thanks,
Mac

Start with the local Consul client behavior described on node q10i22, reproducing a Nomad workload with a Consul Connect proxy and observing ConfigEntry, Intention, DiscoveryChain, and ConnectCA RPCs. Compare the persistent ACL not found errors with the still-valid workload token and xDS stream failures; done means identifying the cause and a reliable fix or confirmed recovery path without restarting the full allocation.

Written by the indexing model from the issue text.