edgeHub: no shared cooldown for a known-dead upstream — each queued operation rebuilds the cloud proxy (874 rebuilds in 1s observed), retaining ~6 KB each

## Summary

On a device whose cellular uplink is detached by the network roughly once a minute, edgeHub rebuilds its
per-client cloud proxy **~874 times in a single second** during each outage window, retaining roughly
6 KB per rebuild. Over 33 h this took edgeHub's cgroup `anon` from ~0 to 230 MiB, against otherwise
identical peer devices sitting at 56–84 MiB after 4.6 days.

The proximate cause looks like **the absence of any shared cooldown for a known-dead upstream**: while
the link is down, each queued operation independently drives a fresh cloud-proxy creation, and there is
nothing that says "the upstream failed 5 ms ago, don't rebuild yet".

**On the EOL question, upfront:** the device runs edgeHub **1.4.43** (`versionInfo.json`:
`build 105359236`, `commit 9f72e9db5e558be4f68345e757e659bf489ef492`, image digest
`sha256:308604711ff15a9324f0fb281bb1d1eaf9e94b57ea27e85555e385eb3db8706d`, built 2024-10-09) with daemon
`aziot-edge 1.5.13`. I know 1.4 is out of support and we are moving to 1.5.x regardless. I am filing
because **the implicated code appears unchanged on `main`** — see the source walk below — so this looks
like a live issue rather than an artefact of an EOL build. Happy to be told otherwise.

## Environment

- edgeHub `mcr.microsoft.com/azureiotedge-hub:1.4` → **1.4.43**; edgeAgent `:1.4`; daemon `aziot-edge 1.5.13`
- Raspberry Pi CM4, 909 MiB RAM, ARM64, Linux containers, kernel 6.1.21-v8+
- edgeHub capped at 450 MiB (`HostConfig.Memory` = `MemorySwap` = 471859200)
- Upstream: LTE via a cellular modem. **The network detaches the PDN session every ~59 s** (confirmed
independently by the carrier's own session records and by ModemManager: `[cm] emm-detached`). Radio is
healthy throughout (RSRP −90 dBm). This is a carrier-side fault we are chasing separately — but it makes
an excellent, if unwelcome, test rig for upstream-flap behaviour.

## Current behaviour

Steady state between outages is ~1 cloud-proxy rebuild per 2 minutes. Then, in one second:

```
95 2026-07-30T07:54:50
874 2026-07-30T07:54:51 <-- 874 × "Client connected to cloud, processing existing subscriptions."
```

The burst begins with:

```
[INF] - "Closing connection for device: , , "
[INF] - Disposing MessagingServiceClient for device Id because of exception -
[INF] - Setting device proxy inactive for device Id
[INF] - Removing device connection for device with removeCloudConnection flag 'True'.
[INF] - Retrying cloud proxy operation SendMessageAsync for .
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Amqp resource is disconnected.
at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSendingLink.SendAmqpMessageAsync(AmqpMessage amqpMessage, TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.SendEventAsync(Message message, TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck)
at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action)
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage)
```

Within that same burst minute there are also **766 × `Skipping for subscription processing, as it is
currently being processed`**, i.e. rebuilds arrive faster than subscription processing completes.

Quantified over a 7.6 h window: **39 burst-minutes containing 34,177 of 34,391 total rebuilds (99.4 %)**,
~5 bursts/hour, ~900 rebuilds each. ~60 network detaches/hour, so only ~8 % of detaches escalate into a
burst — presumably those landing while an operation is in flight.

⚠️ **Burst size grows monotonically over one container lifetime**: 481 rebuilds at 00:35 → 962 at 07:46,
roughly doubling in 7.6 h with no restart. My best guess is the store-and-forward backlog growing, so each
outage has progressively more queued operations to fail — but I have not confirmed that.

### Memory consequence

Two windows, same container, no restart in between:

| window | bursts | rebuilds | cgroup `anon` |
|---|---|---|---|
| 07:41:14 → 07:47:46 (392 s) | 2 | 1,916 | 217.9 → 229.7 MiB (**+11.8 MiB**) |
| 08:04:12 → 08:16:21 (729 s, 13 samples) | **0** | ~6 | **flat, 230.5–233.6 MiB** |

⇒ **~6.3 KB retained per rebuild.** The second window is the control: the 59 s detach cadence continued
throughout it (1 detach/min in all 13 samples) while the heap did not move — so growth tracks *rebuilds*,
not detaches and not elapsed time. Net long-run retention after GC is lower, ~1.4 KB/rebuild (~7 MiB/h).

## Source walk — why I think there is no shared cooldown

From `edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs` on `main`:

```csharp
const int RetryCount = 2;
...
async Task ExecuteOperation(Func> func, string operation)
{
int i = 0;
while (true)
{
ICloudProxy cloudProxy = await this.GetCloudProxy();
try { return await func(cloudProxy); }
catch (Exception e)
{
if (cloudProxy.IsActive) { Events.ThrowExceptionProxyActive(...); throw; }
if (++i == RetryCount) { Events.ThrowExceptionRetryCountReached(...); throw; }
Metrics.AddRetryOperation(this.id, operation);
Events.Retrying(this.id, e, operation); // the log line above
}
}
}

async Task GetCloudProxy()
{
if (!this.innerCloudProxy.IsActive)
{
using (await this.cloudProxyLock.LockAsync())
{
if (!this.innerCloudProxy.IsActive)
{
Events.GettingNewCloudProxy(this.id);
Try cloudProxyTry = await this.cloudProxyGetter();
...
this.innerCloudProxy = cloudProxyTry.Value;
```

Two things stand out:

1. **`RetryCount = 2`, so a single operation retries at most once** — one operation cannot account for 874
rebuilds. The observed rate therefore implies **~874 distinct operations failing in that second**, each
contributing its own rebuild. This is a stampede across operations, not one runaway loop.
2. **`GetCloudProxy()`'s double-checked lock prevents *concurrent* duplicate creation, but not a
*sequential* stampede.** If the freshly created proxy is immediately inactive — which is exactly the
case while the PDN is still down — then each waiter in turn re-enters the lock, sees
`!IsActive`, and creates another proxy. N pending operations produce N proxy creations back to back.

There is **no `Task.Delay`, backoff, or circuit-breaker anywhere in this file** (`grep -ci
'delay|backoff|sleep|circuit'` on `main` → 0). Nothing records "the upstream just failed" in a way that
would let the next operation skip or defer its rebuild attempt.

## Expected behaviour

While the upstream is known to be down, cloud-proxy re-creation should be **rate-limited or
circuit-broken per client**, so that N pending operations during an outage produce O(1) rebuild attempts
with backoff rather than O(N) immediate ones. A short shared cooldown (even 1–2 s) keyed on the last
failed creation would collapse ~900 rebuilds into a handful, and would make the retained-memory cost
proportional to outage *count* rather than to queued-operation count.

## What I have NOT verified

- Whether a mitigation exists **elsewhere** in the connection-establishment path
(`cloudProxyGetter` / `ConnectivityAwareClient` / `ClientProvider`) that should already be damping this.
If so, it is not damping it in practice here, and I would appreciate a pointer.
- The exact `RetryingCloudProxy.cs` contents at our build commit `9f72e9db` — that commit is not resolvable
in the public history, so the source above is `main`. The observed log lines map cleanly onto it.
- Whether 1.5.x behaves differently at runtime. I have not been able to test 1.5.x on this device yet.
- The ~6 KB/rebuild retention is measured from cgroup `anon`, not from a heap dump, so I cannot name the
retained type. I can capture a `dotnet-dump` if that would help — the device sits at ~230 MiB against a
450 MiB cap, so there is headroom.

## Related

- #5201 — same symptom class (unstable link → disconnect/reconnect loop, huge d2c volume), closed as
resolved "somewhere between EdgeAgent/EdgeHub 1.4.6 and 1.4.10". **We are on 1.4.43, so this is not that
regression** — but the burst shape is similar enough that it may be a partial fix or a second path to the
same behaviour.
- #7487, #7497 — also mention `Amqp resource is disconnected` around re-authentication.

## Reproduction

Any uplink that drops the transport every ~60 s while messages are queued should show it. A tc/netem or
iptables rule that blackholes the AMQP endpoint for ~1 s once a minute, with a module publishing steadily,
should reproduce the rebuild stampede without needing a misbehaving cellular network.