Azure / Azure/hpcpack

Could nodes automatically be set offline if they reboot

Open
#45 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
34
Forks
19
Avg merge
41m
Merged PRs (30d)
4

Description

#### Feature Request Description

To be clear: these are not HPC Pack problems, but other windows OS problems that have an impact on our HPC use - and a feature request for HPC Pack that might help us cope with them.

Two problems:-

For various reasons, our cluster is full of Windows 10 Workstation nodes. We are running a fast infiniband application network, but there is a race-condition when Windows 10 boots up, in which SMB does not always start with RDMA enabled, This is solved by restarting the Workstation, HPCSoaDiagMon and Netlogon services - if we don't do that, then the machine effectively locks up forever if file access over the infiniband is attempted.

The second problem is that we have tried **absolutely everything** we can think of to prevent Windows Update from automatically rebooting these workstation nodes monthly, and still it does so - sometimes while jobs are still actually running. This is in our experience the main weakness of Win 10 workstation nodes - (licensing/cost issues prevent us using Win Server as compute nodes)

And the second problem triggers the first, so without us getting any notification, windows update reboots our workstation nodes, and it comes up with RDMA in an inconsistent state, so jobs take forever.

#### Describe Preferred Solution

We would love it if both of these solutions could be fixed upstream - but we don't know how to report them, and have not much confidence in a timely solution. But there is something simple that might be enough:-

If HPC Pack could spot when nodes have gone into the Error state (and have rebooted), and automatically make them offline and not run jobs on them when they return, then we could spot when this has happened, check the node, restart the services if necessary.

#### Describe Alternatives Considered

Windows Update: we are totally lost for options. We've tried Group Policy for the nodes (which are on a domain) set to the "Notify to download / notify to update" options, and we've tried disabling the automatic updates policy altogether - nothing makes any difference; the nodes clearly reboot and we see the Windows Update logs in event viewer. We have no further ideas on that one..

For the recovery, we've tried a Scheduled Task to restart those services - which nearly works, but we found we needed a delay of 1-2 minutes on boot - it varies. If you run the task too early, the restarting doesn't activate RDMA on the infiniband cards; too late, and HPC Pack might already have launched jobs on that node, which then fail if we restart the services, but lock up forever if we leave them.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.