mozilla-releng / mozilla-releng/scriptworker
non-verbose scriptworker should work
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5
- Forks
- 39
- Avg merge
- 4h 18m
- Merged PRs (30d)
- 6
Description
In https://github.com/mozilla-releng/build-puppet/pull/498, I turned off signing scriptworker's verbosity for both the script and daemon. This broke two things:
- scriptworker doesn't log to the screen if
verboseisFalse. This means scriptworker logs stop going to papertrail - nagios relies on the worker.log being up to date to see if the daemon is alive. If the daemon is alive but idle for a long period of time with
verboseset toFalse, the log ages past the warning age.
I turned verbose back on for the daemon in https://github.com/mozilla-releng/build-puppet/pull/500. However, we shouldn't have to keep the daemon verbose.
We should:
-
separate log level from screen output in the config, so we can set the log level to INFO but keep console output for papertrail. (Or, we can add worker.log to the syslog config.)
-
do something to avoid triggering nagios if the log level is higher than DEBUG:
- claimWork at the INFO level
- output an occasional
scriptworker is still aliveevery half hour or so if we've been idle - stop monitoring worker.log in nagios since we haven't really had an issue with hung scriptworker daemons for a long time
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scriptworker/log.py and the claimWork path, then inspect how worker.log and Nagios determine daemon liveness. Review the issue's proposed alternatives and define one approach that preserves Papertrail output, keeps useful log levels, and avoids false Nagios warnings during idle periods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100