Could nodes automatically be set offline if they reboot
- Dominant language
- PowerShell
- Stars
- 34
- Forks
- 19
- Avg merge
- 41m
- Merged PRs (30d)
- 4
Description
#### Feature Request Description
To be clear: these are not HPC Pack problems, but other windows OS problems that have an impact on our HPC use - and a feature request for HPC Pack that might help us cope with them.
Two problems:-
For various reasons, our cluster is full of Windows 10 Workstation nodes. We are running a fast infiniband application network, but there is a race-condition when Windows 10 boots up, in which SMB does not always start with RDMA enabled, This is solved by restarting the Workstation, HPCSoaDiagMon and Netlogon services - if we don't do that, then the machine effectively locks up forever if file access over the infiniband is attempted.
The second problem is that we have tried **absolutely everything** we can think of to prevent Windows Update from automatically rebooting these workstation nodes monthly, and still it does so - sometimes while jobs are still actually running. This is in our experience the main weakness of Win 10 workstation nodes - (licensing/cost issues prevent us using Win Server as compute nodes)
And the second problem triggers the first, so without us getting any notification, windows update reboots our workstation nodes, and it comes up with RDMA in an inconsistent state, so jobs take forever.
#### Describe Preferred Solution
We would love it if both of these solutions could be fixed upstream - but we don't know how to report them, and have not much confidence in a timely solution. But there is something simple that might be enough:-
If HPC Pack could spot when nodes have gone into the Error state (and have rebooted), and automatically make them offline and not run jobs on them when they return, then we could spot when this has happened, check the node, restart the services if necessary.
#### Describe Alternatives Considered
Windows Update: we are totally lost for options. We've tried Group Policy for the nodes (which are on a domain) set to the "Notify to download / notify to update" options, and we've tried disabling the automatic updates policy altogether - nothing makes any difference; the nodes clearly reboot and we see the Windows Update logs in event viewer. We have no further ideas on that one..
For the recovery, we've tried a Scheduled Task to restart those services - which nearly works, but we found we needed a delay of 1-2 minutes on boot - it varies. If you run the task too early, the restarting doesn't activate RDMA on the infiniband cards; too late, and HPC Pack might already have launched jobs on that node, which then fail if we restart the services, but lock up forever if we leave them.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.