theupdateframework / theupdateframework/python-tuf
logging in test runner / CI : higher log level but only print failing tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 304
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 17
Description
We should do what in-toto does: have the test runner output much more logging but only for failing tests. This should be especially useful for CI but also for developers.
Assuming we hard-code a good log level (see #1093 for potential alternatives), I think we only need roughly this in aggregate_tests.py:
class TestHandler(logging.StreamHandler):
def __init__(self):
super().__init__(self)
@property
def stream(self):
return sys.stderr
@stream.setter
def stream(self, value):
pass
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO, handlers=[TestHandler()])
# current testrunner code here...
This is because the testrunner buffering will not work with long lived loggers otherwise: the default handler caches sys.stderr value and testrunner wants to modify it between test runs (and buffering is required to only print the failing test output).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read aggregate_tests.py and the test-runner buffering behavior described in the issue; compare the proposed handler with the in-toto approach. Confirm that higher-level logging is available for failing tests while passing-test output remains buffered, including in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100