Report live mount and backend health from the storage proxy

The storage proxy cannot currently tell whether a volume is actually usable. Listing volumes echoes the configuration file, so a dropped mount is indistinguishable from a healthy one, and the hardware information call is hardcoded to report healthy for the local filesystem backends.

Probes

Probe the mount and the backend appliance separately, because they fail independently: a vendor appliance can answer its management API while a network mount has silently dropped.

Compare the device id against the value captured when the volume was initialized, to catch a mount that fell off and left the path pointing at the directory underneath it. The baseline lives on the volume itself and cannot survive a restart, because a device number is assigned by the kernel at mount time.

Query filesystem statistics in an executor under a timeout, so that a dead network mount surfaces as a timeout instead of hanging the caller.

Read a marker file holding the volume name, to catch a path that is serving different storage than declared. The marker is the only check that survives a restart, because it lives on the storage itself. The proxy never writes it; an operator creates it with a CLI command, and an existing marker is never overwritten. An absent marker means identity is unverified, not broken, so that installations upgrading into this feature do not all report unhealthy. A marker that was present when the proxy started and is gone later is a remount signal and is reported as unhealthy.

Implement the hardware information call for the local filesystem backends so that appliance health is real rather than assumed. For a local filesystem the appliance is the mount itself, so the call translates the latest mount probe result instead of running a probe of its own inside a request.

Execution model

A periodic observer probes each volume and each backend and keeps the latest result in memory. The heartbeat loop reads that memory and ships it. Probe cadence and heartbeat cadence stay independent, so a slow or hung probe never delays a heartbeat and never makes a healthy proxy look dead.

Each probe runs as its own periodic task on the local cron the proxy already uses for its heartbeat: one task per volume, so per-volume independence and staggering come from the runner rather than from a hand-rolled fan-out inside a single observer.

Every reported entry carries the time it was last checked. The proxy models no separate unknown state; the manager decides healthy from stale using that timestamp.

A volume that has never been probed is still declared, with no check time, so that the manager can tell it apart from a volume this proxy does not serve at all.

Probes run independently per volume. A dead network mount blocks in its system call, and walking the volumes in a single loop would let one of them starve all the others.

A probe that timed out must not be retried while the previous attempt is still outstanding. Cancelling the await does not release the executor thread, which stays blocked in the system call, so repeated retries would accumulate threads. Mark the volume as hung instead and wait for the outstanding attempt to finish. The probes use a dedicated thread pool rather than the default executor, so that a blocked thread cannot starve unrelated file operations.

Probes never run synchronously inside a request, where one hung mount would stall the endpoint.

Results travel in the heartbeat event. No separate health event type is introduced.

Design: BEP-1078 (Storage Proxy Enhancement).

JIRA Issue: BA-7654

Start by locating the storage proxy's existing heartbeat loop and local cron runner, then trace how volume and backend information currently reaches heartbeat events. Define the independent per-volume probes, timeout and dedicated executor behavior, marker handling, and in-memory timestamps described here. Done means heartbeat reports mount and appliance health without blocking on a hung volume, including never-probed volumes.

Written by the indexing model from the issue text.