pytest-dev / pytest-dev/pytest

Cannot use python logging without maintaining all records in caplog

Open
#8,307 13 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: logging type: performance
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Description

I'd like to be able to use python's built-in logging, specifically to stream logging records to CLI and file, without having all log records stored in memory in the caplog fixture.

The reason I don't want the records to be stored in caplog is that this is essentially a forced memory leak. In the use-case where I discovered this, all DEBUG-level logging records were being stored persistently in caplog even though my CLI logging level was INFO, file handler level was DEBUG. The records were being streamed to file as expected, but I don't want them kept in memory. This was a very long-running test, the accumulating DEBUG records eventually resulted in a memory error.

Maybe I just don't know how to use caplog/logging in such a way that I can handle my logging streams as desired without keeping the log records in memory. If that's the case, would it be possible to get that clarification?

Environment


Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)]

system='Windows', release='10', version='10.0.16299', machine='AMD64', processor='Intel64 Family 6 Model 142 Stepping 10, GenuineIntel'

Package            Version
------------------ -------
atomicwrites       1.4.0
attrs              20.3.0
colorama           0.4.4
importlib-metadata 3.4.0
iniconfig          1.1.1
packaging          20.9
pip                20.3.3
pluggy             0.13.1
py                 1.10.0
pyparsing          2.4.7
pytest             6.2.2
setuptools         51.0.0
toml               0.10.2
typing-extensions  3.7.4.3
wheel              0.36.2
zipp               3.4.0

Example

import logging
logger = logging.getLogger(__name__)
test_logging_ex(caplog):
    assert not caplog.records
    logger.info("I'd like to see this message in the CLI, but I can't"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided caplog example with Python logging and inspect how the caplog fixture retains records while CLI and file handlers process them. Done means logging can be streamed to the requested destinations without retaining all records in memory, with behavior covered for the long-running test case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.