Configurable agent-scan retries and maintenance-aware VM resurrection

Context

During a recent investigation of a BOSH-managed Cloud Foundry deployment on Azure, a diego-cell VM was recreated shortly after a live-migration event. We have not established that the migration caused BOSH's recovery decision, so this is a resilience feature request targeting IaaS-specific events.

Azure's public documentation notes that during a live migration some workloads can experience temporary "performance degradation in the few minutes leading up to the VM pause". More generally, platform maintenance or transient connectivity problems can delay communication with an otherwise recoverable VM.

The goal of this feature request is to tolerate bounded, transient agent unresponsiveness without substantially increasing failure-detection time for every VM. While it is possible to increase the properties.hm.intervals.agent_timeout, this would delay issue detection and potential counter-measures of the director in the event of a VM downtime that is not related to IaaS maintenance events.

Existing behavior

In the referenced upstream revision:

• ProblemScanner::VmScanStage uses a 10-second agent RPC timeout and explicitly sets get_state retries to zero.
• ProblemHandlers::UnresponsiveAgent already pings the agent before recreation and skips the resolution if it responds.

These Director checks are separate from the Health Monitor's configurable heartbeat timeout. The proposals below extend the existing safeguards; they do not assume that BOSH immediately deletes a VM after one missed heartbeat.

Proposal 1: Configurable, bounded retries in the VM scanner

Allow operators to configure the scanner's agent RPC timeout, retry count, and retry delay/backoff instead of requiring a code change.

• Reuse the existing agent-client retry mechanism where possible, with an explicit maximum elapsed budget.
• Retry appropriate transient failures of read-only agent calls, rather than permanent errors or arbitrary operations.
• Stop retrying once the agent responds and retain the existing final validation before destructive recovery.
• Preserve the distinction between a missing VM and an existing VM whose agent is temporarily unreachable.
• Keep the current defaults for backward compatibility, and log attempts, elapsed time, and the final classification so operators can tune the policy.

This would provide an incremental improvement without requiring provider-specific maintenance integration or a blanket increase in global heartbeat timeouts.

Proposal 2: Provider-neutral maintenance awareness, starting with Azure Scheduled Events

This is my preferred longer-term direction: keep normal failure detection fast, but use explicit maintenance context to inform recovery decisions for the affected instance.

Azure Scheduled Events could be an initial provider implementation. A provider-neutral mechanism could relay normalized maintenance context to BOSH, allowing the Director/Health Monitor to distinguish an expected maintenance window from unexplained agent unresponsiveness.

Important requirements:

• Make the integration opt-in and scope any recovery deferral to the affected instance, not the entire deployment.
• Continue health monitoring and alerting while applying a bounded grace period to destructive recovery.
• Preserve fresh agent validation and normal handling of genuinely missing or persistently unresponsive VMs.
• Expire protection automatically. Stale or repeated events, an unavailable provider endpoint, or a failed integration must not disable recovery indefinitely.
• Account for event cancellation, unknown duration, and missing or late notifications. Do not assume that a notification always precedes every period of guest impairment.
• Keep provider-specific discovery and event semantics in adapters rather than embedding Azure-specific behavior throughout the generic recovery path.

The integration point needs design discussion. For example, Azure's IMDS endpoint is guest-local, so an agent/platform component or an external integration may need to collect and relay the information. A CPI-only implementation may not be sufficient.

Expected outcome

Recoverable agent interruptions should be less likely to result in unnecessary VM recreation, while genuine failures remain recoverable within a bounded, operator-understood time.

The configurable-retry improvement and maintenance-awareness design could progress independently. Feedback on the configuration surface for the first proposal and an appropriate extension point for the second would be welcome.

Additional References

• scheduled-events-mock-server could be helpful with Proposal 2

/cc @a-hassanin

Start with src/bosh-director/lib/bosh/director/problem_scanner/vm_scan_stage.rb and src/bosh-director/lib/bosh/director/problem_handlers/unresponsive_agent.rb, then inspect the existing agent-client retry mechanism and related configuration. Done requires a scoped design for bounded configurable retries and/or provider-neutral maintenance context, including defaults, expiry, logging, and validation behavior; the issue does not name tests or a single implementation entry point.