Avoid puppetserver failure after automatic JRE package updates

After a minor package update is released to Debian, puppetserver can sometimes get into a stuck state, which typically looks like this on the side of individual server clients:

krinkle@wp-03:~$ sudo run-puppet-agent
Info: Using environment 'staging'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error:
Exception while executing '/etc/puppet/code/environments/staging/bin/config-version.sh': Cannot run program 

The fix is simply to restart the puppet server:

krinkle@puppet-04:~$ sudo systemctl status puppetserver

Details about the root cause in the comments at https://github.com/jquery/infrastructure-puppet/commit/c08df0271ed84068ea50ff520bd2ac7a943961e4.

@supertassu wrote
At Wikimedia we disabled automatic JRE updates on Puppetservers for this exact reason. We could do that here, but I would prefer to avoid this option as our server is open to the internet. (I guess that'd be relatively trivial to fix, except figuring out how to provision new nodes.) The other option is to hook up some logic to automatically restart the puppetserver service when needed and figure out the systemd magic to run that after the automatic updates finish.

See also:

• https://phabricator.wikimedia.org/T377803
• http://phabricator.wikimedia.org/T385553
• gerrit.wikimedia.org/g/operations/puppet.git:/modules/profile/manifests/puppetserver/wmcs.pp

Start with the puppetserver systemd service and the referenced modules/profile/manifests/puppetserver/wmcs.pp configuration, then review how Debian automatic package updates finish. Define and implement the restart behavior there, and verify with systemctl status puppetserver and a successful run-puppet-agent invocation after a JRE update.