mozilla-releng / mozilla-releng/scriptworker

non-verbose scriptworker should work

Open
#341 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
39
Avg merge
4h 18m
Merged PRs (30d)
6

Description

In https://github.com/mozilla-releng/build-puppet/pull/498, I turned off signing scriptworker's verbosity for both the script and daemon. This broke two things:

  • scriptworker doesn't log to the screen if verbose is False. This means scriptworker logs stop going to papertrail
  • nagios relies on the worker.log being up to date to see if the daemon is alive. If the daemon is alive but idle for a long period of time with verbose set to False, the log ages past the warning age.

I turned verbose back on for the daemon in https://github.com/mozilla-releng/build-puppet/pull/500. However, we shouldn't have to keep the daemon verbose.

We should:

  • separate log level from screen output in the config, so we can set the log level to INFO but keep console output for papertrail. (Or, we can add worker.log to the syslog config.)

  • do something to avoid triggering nagios if the log level is higher than DEBUG:

    • claimWork at the INFO level
    • output an occasional scriptworker is still alive every half hour or so if we've been idle
    • stop monitoring worker.log in nagios since we haven't really had an issue with hung scriptworker daemons for a long time

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scriptworker/log.py and the claimWork path, then inspect how worker.log and Nagios determine daemon liveness. Review the issue's proposed alternatives and define one approach that preserves Papertrail output, keeps useful log levels, and avoids false Nagios warnings during idle periods.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.