Should we disable auto-update of system packages?

Originally posted by @richardlau in https://github.com/nodejs/build/issues/3754#issuecomment-2152673030

Should we disable auto-update of system packages?

That's a good question. On the one hand it would mean no unexpected surprises but the onus is then back on us to keep the systems reasonably up-to-date. Maybe we should schedule regular (monthly?) maintenance windows for applying routine package updates?

@nodejs/build-infra thoughts?

This manifests as Jenkins being unable to spawn git/bash after an automatic Java Runtime Environment update until the Jenkins agent is restarted, e.g.

09:26:33 ERROR: Error cloning remote repo 'origin'
09:26:33 hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit
09:26:33 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1073)
...
09:26:33 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit"): error=0, Failed to exec spawn helper: pid: 1881683, exit value: 1
09:26:33 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
...
09:26:33 Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 1881683, exit value: 1
09:26:33 	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)

This has affected the Ubuntu machines:

• https://github.com/nodejs/build/issues/3754
• https://github.com/nodejs/build/issues/3643

and also Debian:

• https://github.com/nodejs/build/issues/3941
• https://github.com/nodejs/build/issues/3625

Alternatively maybe there's a way to have the Systemd service running the Jenkins agent to restart if Java gets updated.

Start by reviewing the linked Ubuntu and Debian incidents and the systemd service running the Jenkins agent. Compare the options of disabling automatic package updates, scheduling maintenance windows, or restarting the agent after Java updates. Done means the chosen policy is documented and implemented, with agent recovery verified after a runtime update.