theupdateframework / theupdateframework/python-tuf

logging in test runner / CI : higher log level but only print failing tests

Open
#1,120 3 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

testing
Dominant language
Python
Stars
1.7k
Forks
304
Avg merge
1d 2h
Merged PRs (30d)
17

Description

We should do what in-toto does: have the test runner output much more logging but only for failing tests. This should be especially useful for CI but also for developers.

Assuming we hard-code a good log level (see #1093 for potential alternatives), I think we only need roughly this in aggregate_tests.py:

class TestHandler(logging.StreamHandler):
  def __init__(self):
    super().__init__(self)

  @property
  def stream(self):
    return sys.stderr

  @stream.setter
  def stream(self, value):
    pass

if __name__ == '__main__':
  logging.basicConfig(level=logging.INFO, handlers=[TestHandler()])
  # current testrunner code here...

This is because the testrunner buffering will not work with long lived loggers otherwise: the default handler caches sys.stderr value and testrunner wants to modify it between test runs (and buffering is required to only print the failing test output).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read aggregate_tests.py and the test-runner buffering behavior described in the issue; compare the proposed handler with the in-toto approach. Confirm that higher-level logging is available for failing tests while passing-test output remains buffered, including in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.