Discussion: Separate jenkins logs from the main syslog

Spotted after identifying https://github.com/nodejs/build/issues/4434

At some point we must have moved the jenkins log files into the syslog. There are files in /var/log/jenkins from August 2022 (Four years ago) but no more and the jenkins information has been put into the syslog where it is fighting for visibility among all the firewall activity (which also goes to kern.log so we have that duplicated). As an example in the last full week of syslog activity we had 1,681,013 entries from jenkins and 25,234,966 from the firewall out of a total of 27,074,871

I would propose that we look at moving it out of the syslog into standalone files which are a bit easier to process when needed. While I appreciate that grep jenkins in the syslog can also work I feel that having it in separate rotated files would just be a bit cleaner.

I had a brief discussion with Richard on this last week but does anyone recall if there was a driving reason for moving towards using the syslog? There's nothing obvious I can see in the issue history in this repository about that change.

Related: https://github.com/nodejs/build/issues/4419

Review the current logging setup that sends Jenkins entries into syslog, comparing /var/log/jenkins with syslog and kern.log. Check the context in issues 4434 and 4419 and confirm why the change was made. Done means Jenkins output is handled in separate rotated files without losing required logging or visibility.