pkgutil direct use of zipimport._zip_directory_cache is no longer safe
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
pkgutil directly uses zipimport._zip_directory_cache, in this block:
def iter_zipimport_modules(importer, prefix=''):
dirlist = sorted(zipimport._zip_directory_cache[importer.archive])
since https://github.com/python/cpython/pull/103208 , this is no longer safe, because that made zipimport no longer always refresh the cache on invalidation. I think that if the cache has been invalidated but not accessed 'normally' since then, this direct access attempt can cause a crash as it doesn't trigger a refresh:
E KeyError: '/tmp/dask-scratch-space/scheduler-belyj1dp/myfile.zip'
I observed this in the test suite for dask-distributed - see https://github.com/dask/distributed/issues/8708 - but it seems like a general issue, and @desmondcheongzx asked me to file an issue, so here it is. That test suite hits it in a test that runs async and creates archives under the same name repeatedly, but there are probably other ways to hit this, I guess.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
- gh-121705
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkgutil.iter_zipimport_modules and the direct access to zipimport._zip_directory_cache described in the issue; compare it with the cache invalidation behavior introduced by CPython PR 103208. Reproduce the repeated-archive scenario from dask-distributed issue 8708 and verify that the resulting KeyError no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100