python / python/cpython

MemoryHandler can't be setup through a DictConfig if any other logger is setup after it

Open
#123,239 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

This is the logging that's failing.

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "fine-logger1": {
            'level': logging.INFO,
            'formatter': 'verbose-console',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',
        },
        "fine-logger2": {
            "level": logging.INFO,
            "class": "logging.FileHandler",
            "filename": Path("logs/somefilename.log"),
            "mode": "a",
            "formatter": "verbose",
        },
        "fine-logger3": {
            "level": logging.DEBUG,
            "class": "logging.FileHandler",
            "filename": Path("logs/errors/somefilename.log"),
            "delay": True,
            "formatter": "verbose",
        },
        "bugged-logger3": {
            "level": logging.DEBUG,
            "class": "logging.handlers.MemoryHandler",
            "capacity": 9999999,
            "flushLevel": logging.ERROR,
            "target": "fine-logger3",
            "flushOnClose": False,
        },
    },
    "loggers": {
        "bug-logger": {
            "handlers": ["fine-logger1", "fine-logger2", "bugged-logger3"],
            "level": logging.DEBUG,
            "propagate": True
        },
    },
}

After it's been setup, if any other handler is setup afterwards (in my case it's the copernicusmarine logger) the handler bugged-logger3 will have a None inside its target field, as every handler that's not incremental will delete existing unused handlers.

I've dug into the source code of logging and I think I've tracked down where this is happpening, but I'm not really sure on how to fix it.
In the config file python3.11/logging/config.py in the configure() function there's a call to _clearExistingHandlers():

            else:
                disable_existing = config.pop('disable_existing_loggers', True)

                _clearExistingHandlers() <-----

                # Do formatters first - they don't refer to anything else
                formatters = config.get('formatters', EMPTY_DICT)
                for name in formatters:
                    try:
                        formatters[name] = self.configure_formatter(
                                                            formatters[name])

If we comment this line, it works completely fine. But I guess this just keeps handlers loaded in memory and that's not a proper solution.

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python3.11/logging/config.py, especially DictConfig.configure() and _clearExistingHandlers(). Reproduce the provided configuration, then configure another logger and inspect whether MemoryHandler.target becomes None. Done means later non-incremental configuration no longer breaks the MemoryHandler target while handler cleanup still behaves correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.