drakkar-lig / drakkar-lig/walt-python-packages

Enable boot timeout monitoring on Grid5000

Open
#130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
4
No merged PRs in 30d

Description

Most Grid5000 servers are very slow to reboot, so the boot.timeout setting of the nodes should be updated to a larger value when the walt platform is deployed.
For instance, on the following trace, chifflot-2 took 4min15 to boot, although it is on the same site as the server (at Lille).
Six minutes is probably an appropriate value (the defaut boot.timeout is 3min).

In the trace, we see that hard reboots do not work when the timeout is reached (which is a good thing until we increase this timeout) because we only support hard reboot using PoE.
However, in this case taurus-1 did fail to boot properly and I used walt node reboot --hard taurus-1 in another terminal to reboot it, which eventually allowed it to boot properly. In this case (G5K plugin), the hard reboot relies on command kareboot3.
So in this case of reaching boot.timeout (6 minutes) on G5K, we should allow using this path based on kareboot3 for hard-rebooting, since we don't have access on PoE switches.

Jun 25 08:34:28 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI set_image ('all-nodes', 'pc-x86-64-default') {}
Jun 25 08:34:29 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI wait_for_nodes ('all-nodes',) {}
Jun 25 08:37:09 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI sync_clock () {}
Jun 25 08:37:22 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI sync_clock () {}
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: chifflot-4: boot timeout reached, trying hard-reboot (8 retries left).
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: Node chifflot-4: failed poe-reboot (unknown lldp network position)
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: chifflot-2: boot timeout reached, trying hard-reboot (8 retries left).
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: Node chifflot-2: failed poe-reboot (unknown lldp network position)
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: taurus-1: boot timeout reached, trying hard-reboot (8 retries left).
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: Node taurus-1: failed poe-reboot (unknown lldp network position)
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: taurus-10: boot timeout reached, trying hard-reboot (8 retries left).
Jun 25 08:37:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: Node taurus-10: failed poe-reboot (unknown lldp network position)
Jun 25 08:37:24 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: node taurus-10 is booted
Jun 25 08:37:25 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: node chifflot-4 is booted
Jun 25 08:37:45 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI report_lldp_neighbor ('00:01:e8:8b:41:81', 'TenGigabitEthernet 0>
Jun 25 08:38:35 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI sync_clock () {}
Jun 25 08:38:38 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: node chifflot-2 is booted
Jun 25 08:38:54 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI report_lldp_neighbor ('a0:3d:6f:7f:96:69', 'Ethernet1/2') {}
Jun 25 08:39:17 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI shell_autocomplete ('walt-g5k', ['NODE', 'walt', 'node', 'shell'>
Jun 25 08:39:21 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI shell_autocomplete ('walt-g5k', ['NODE', 'walt', 'node', 'shell'>
Jun 25 08:39:22 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI shell_autocomplete ('walt-g5k', ['NODE', 'walt', 'node', 'shell'>
Jun 25 08:39:24 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI shell_autocomplete ('walt-g5k', ['NODE', 'walt', 'node', 'shell'>
Jun 25 08:39:25 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI filter_ownership ('chifflot-2',) {}
Jun 25 08:39:25 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI get_nodes_ip ('chifflot-2',) {}
Jun 25 08:39:25 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI wait_for_nodes ('chifflot-2',) {}
Jun 25 08:39:59 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI shell_autocomplete ('walt-g5k', ['NODE', 'walt', 'node', 'shell'>
Jun 25 08:40:01 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI filter_ownership ('taurus-10',) {}
Jun 25 08:40:01 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI get_nodes_ip ('taurus-10',) {}
Jun 25 08:40:01 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI wait_for_nodes ('taurus-10',) {}
Jun 25 08:40:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: taurus-1: boot timeout reached, trying hard-reboot (7 retries left).
Jun 25 08:40:23 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: Node taurus-1: failed poe-reboot (unknown lldp network position)
Jun 25 08:42:16 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI show_nodes ('eduble', False, False) {}
Jun 25 08:42:24 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI show_nodes ('eduble', True, False) {}
Jun 25 08:43:17 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI filter_ownership ('taurus-1',) {}
Jun 25 08:43:17 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI reboot_nodes ('taurus-1',) {'hard_only': True}
Jun 25 08:45:57 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: NSAPI sync_clock () {}
Jun 25 08:46:02 chiclet-2.lille.grid5000.fr walt-server-daemon[3800]: node taurus-1 is booted
Jun 25 08:46:18 chiclet-2.lille.grid5000.fr walt-server-daemon[3802]: hub api_call: CSAPI show_nodes ('walt-g5k', False, False) {}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the handling of the boot.timeout setting and the Grid5000 plugin's kareboot3 hard-reboot path. Check how timeout-triggered reboots currently select PoE, then verify that Grid5000 deployments use a six-minute timeout and can fall back to kareboot3 when the timeout is reached.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.