pytest-dev / pytest-dev/pytest

caplog fixture should not be affected by global log level

Open
#7,162 11 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: logging type: proposal
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

#7159 made me realize something: I think caplog by default should not be affected by the global log level.

For example:

import logging

def test(caplog):
    logging.info("info")
    logging.critical("critical")
    print(caplog.messages)
    assert 0

Running:

 λ pytest .tmp\test-caplevel.py
======================== test session starts ========================
collected 1 item

.tmp\test-caplevel.py F                                        [100%]

============================= FAILURES ==============================
_______________________________ test ________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x000001827FB75358>

    def test(caplog):
        logging.info("info")
        logging.critical("critical")
        logging.debug("debug")
        print(caplog.messages)
>       assert 0
E       assert 0

.tmp\test-caplevel.py:8: AssertionError
----------------------- Captured stdout call ------------------------
['critical']
------------------------- Captured log call -------------------------
CRITICAL root:test-caplevel.py:5 critical
========================= 1 failed in 0.25s =========================
λ pytest .tmp\test-caplevel.py -o log_level=INFO
======================== test session starts ========================
platform win32 -- Python 3.6.6, pytest-5.4.1.dev154+gbe6849644.d20200501, py-1.8.1, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('d:\\projects\\pytest\\.hypothesis\\examples')
rootdir: d:\projects\pytest\.tmp, inifile: pytest.ini
plugins: hypothesis-4.36.0, forked-1.1.1, xdist-1.31.0
collected 1 item

.tmp\test-caplevel.py F                                        [100%]

============================= FAILURES ==============================
_______________________________ test ________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x0000014A1CCD32E8>

    def test(caplog):
        logging.info("info")
        logging.critical("critical")
        logging.debug("debug")
        print(caplog.messages)
>       assert 0
E       assert 0

.tmp\test-caplevel.py:8: AssertionError
----------------------- Captured stdout call ------------------------
['info', 'critical']
------------------------- Captured log call -------------------------
INFO     root:test-caplevel.py:4 info
CRITICAL root:test-caplevel.py:5 critical
====================== short test summary info ======================
FAILED .tmp\test-caplevel.py::test - assert 0
========================= 1 failed in 0.24s =========================

So depending of the loglevel setting, the test might fail. #7159 is a step in the right direction, because calling caplog.set_level will overwrite the global log level.

But I think this is kind of error prone too, and caplog should have a default log-level value (say INFO), independent from the global log level, which is changed only by calling set_level explicitly.

cc @thisch @ruaridhw

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the caplog fixture and its set_level entry point, then reproduce the two command-line examples from the issue. Done means caplog uses a stable default level independently of the global log level, while an explicit set_level call still changes the captured level.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.