saltstack / saltstack/salt

[BUG] KeyError raised in LazyDict's _getitem__ happy path (in LazyLoader)

Open
#59,690 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug lazy-loader severity-low
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
There can be a KeyError raised in the marked line below, even though this is the happy path:

class LazyDict(MutableMapping):
    def __getitem__(self, key):
        """
        Check if the key is ttld out, then do the get
        """
        if self._missing(key):
            raise KeyError(key)

        if key not in self._dict and not self.loaded:
            # load the item
            if self._load(key):
                log.debug("LazyLoaded %s", key)
                return self._dict[key]   <--------- KeyError
            else:
                log.debug(
                    "Could not LazyLoad %s: %s", key, self.missing_fun_string(key)
                )
                raise KeyError(key)
        else:
            return self._dict[key]

This can happen in LazyLoader(inherits from LazyDict) due to the following circumstances:

  1. _missing(key) is not implemented in LazyLoader and uses the "default" from LazyDict (return False)
  2. self._load(key) (implemented in LazyLoader) returns True when a key is in self.missing_modules

These two conditions allow the following situation:

  1. LazyLoader.__getitem__(foo) is called
  2. LazyLoader calls super().__getitem__(foo)
  3. LazyDict.__getitem__(foo) checks that foo is not loaded yet
  4. LazyDict.__getitem__(foo) calls self._load(foo)
  5. LazyLoader._load(foo) tries to load "foo" via LazyLoader._load_module(foo)
  6. LazyLoader._load_module(foo) can't load "foo" and puts it self.missing_modules
  7. LazyLoader._load(foo) returns None
  8. LazyDict.__getitem__(foo) logs "Could not LazyLoad ..." and raises KeyError(foo)
  9. For a second time LazyLoader.__getitem__(foo) is called
  10. Which uses super().__getitem__(foo) again, leading to
  11. LazyDict.__getitem__(foo) calls self._load(foo)
  12. This time, LazyLoader._load(foo) looks at self.missing_modules and returns True
  13. LazyDict.__getitem__(foo) is happy, logs "LazyLoaded foo"
  14. KeyError

Expected behavior
The KeyError should be raised either in the else branch or even earlier when self._missing(key) is called. I think the "proper" way is to define self._missing() in LazyLoader and remove the x in self.missing_module checks.

I would like to hear your opinions on the best way to fix this, I don't mind fixing the bug once I know which path we want.

Versions Report
The links to the methods that cause the issue are from the latest commit in master at the time of writing this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading LazyDict.getitem in salt/utils/lazy.py and LazyLoader.getitem, _load, and _load_module in salt/loader.py. Trace the missing_modules path described in the issue and determine how an already-known missing key should consistently produce KeyError; done means the happy path no longer indexes an absent key and the behavior is covered by the relevant loader tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.