[BUG] KeyError raised in LazyDict's _getitem__ happy path (in LazyLoader)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
There can be a KeyError raised in the marked line below, even though this is the happy path:
class LazyDict(MutableMapping):
def __getitem__(self, key):
"""
Check if the key is ttld out, then do the get
"""
if self._missing(key):
raise KeyError(key)
if key not in self._dict and not self.loaded:
# load the item
if self._load(key):
log.debug("LazyLoaded %s", key)
return self._dict[key] <--------- KeyError
else:
log.debug(
"Could not LazyLoad %s: %s", key, self.missing_fun_string(key)
)
raise KeyError(key)
else:
return self._dict[key]
This can happen in LazyLoader(inherits from LazyDict) due to the following circumstances:
_missing(key)is not implemented inLazyLoaderand uses the "default" fromLazyDict(return False)self._load(key)(implemented inLazyLoader) returnsTruewhen a key is inself.missing_modules
These two conditions allow the following situation:
LazyLoader.__getitem__(foo)is calledLazyLoadercallssuper().__getitem__(foo)LazyDict.__getitem__(foo)checks thatfoois not loaded yetLazyDict.__getitem__(foo)callsself._load(foo)LazyLoader._load(foo)tries to load "foo" viaLazyLoader._load_module(foo)LazyLoader._load_module(foo)can't load "foo" and puts itself.missing_modulesLazyLoader._load(foo)returnsNoneLazyDict.__getitem__(foo)logs "Could not LazyLoad ..." and raisesKeyError(foo)- For a second time
LazyLoader.__getitem__(foo)is called - Which uses
super().__getitem__(foo)again, leading to LazyDict.__getitem__(foo)callsself._load(foo)- This time,
LazyLoader._load(foo)looks atself.missing_modulesand returnsTrue LazyDict.__getitem__(foo)is happy, logs "LazyLoaded foo"KeyError
Expected behavior
The KeyError should be raised either in the else branch or even earlier when self._missing(key) is called. I think the "proper" way is to define self._missing() in LazyLoader and remove the x in self.missing_module checks.
I would like to hear your opinions on the best way to fix this, I don't mind fixing the bug once I know which path we want.
Versions Report
The links to the methods that cause the issue are from the latest commit in master at the time of writing this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading LazyDict.getitem in salt/utils/lazy.py and LazyLoader.getitem, _load, and _load_module in salt/loader.py. Trace the missing_modules path described in the issue and determine how an already-known missing key should consistently produce KeyError; done means the happy path no longer indexes an absent key and the behavior is covered by the relevant loader tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100