Liveliness Probe/Health-check and Automatic Retrieval

Requirements for Manageability – Metro AI Suite

Background
The Device Management Toolkit (DMT) provides out-of-band (OOB) management of Intel AMT-capable devices via the Management Presence Server (MPS), RPS, and the Sample Web UI. Operators can currently issue on-demand power actions (power on/off/cycle/reset), retrieve hardware inventory, and open KVM/SOL sessions through DMT console.
However, DMT today has no built-in mechanism to:
• Continuously verify that a managed edge node's operating system / application stack is alive and responsive (as opposed to merely being powered on at the AMT firmware level).
• Automatically trigger a corrective power action when a node is determined to be unhealthy.
For the transportation sector — where thousands of edge nodes are deployed in vehicles, roadside units (RSUs), depots, tunnels, level crossings, and remote signalling cabinets — manual triage is operationally infeasible. A frozen OS for a smart intersection app or smart tolling or smart parking, a hung RSU, or an unresponsive fleet telematics gateway currently requires a field technician dispatch even though AMT could remotely recover the node in seconds.

Problem Statement:
Large-scale transportation deployments require a hands-off, closed-loop recovery mechanism in DMT that:

1. Detects when a managed device stops responding at the OS/application layer while AMT is still reachable.
2. Escalates through a configurable remediation ladder (soft → hard) using existing AMT power capabilities.
3. Records every decision for audit, SLA reporting, and regulatory compliance (e.g., EN 50155, ISO 26262 process artifacts) in events.

Functional Requirements
ID Requirement
1 DMT support configurable liveness probes per device.
2 Supported probe types include: ICMP ping, TCP connect, HTTP(S) GET with expected status/body
3 Each probe define interval, timeout, failure-threshold (N consecutive failures), success-threshold (for recovery)
4 DMT distinguish OS-unresponsive but AMT-reachable from fully offline (AMT unreachable) and act differently in each state.
5 On breach of failure-threshold, DMT execute a remediation policy — an ordered ladder of AMT power actions with wait windows between steps.
6 Operators be able to pause, override, or acknowledge auto-remediation on a device.

Transportation-Sector Use Cases
Roadside V2X unit — Agent heartbeat missing for 60-90 sec while AMT still reachable, appropriate power reset issued; event streamed to the DMT console event section.

Start by reading the Sample WebUI integration points for DMT device management and the console event section, then trace how existing power actions and inventory retrieval are exposed. The work is complete when configurable probes, remediation controls, device-state distinction, and auditable events are supported across the described requirements.