Improve performance of `os.walk()`
Open
Nobody has claimed this yet.
performance
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
There are a couple of minor performance improvements possible in os.walk():
- We don't need to manually pump the
os.scandiriterator, given we handle exceptions fromnext()like exceptions fromscandir()itself, i.e. by ignoring the problematic directory and moving on. We can use aforloop like filthy casuals. In bottom-up mode, we can handle exceptions fromentry.is_symlink()in the same block as those fromentry.is_dir(), which avoids a few temporary variables.- In top-down mode, we can call
os.path.join()once on a parent directory rather than for each child path.
Linked PRs
- gh-119186
- gh-119473
- gh-119573
- gh-119750
- gh-121431
- gh-121432
- gh-121433
- gh-121435
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the os.walk() entry point and inspect its use of os.scandir(), entry.is_dir(), entry.is_symlink(), and os.path.join(). Review linked PRs gh-119186, gh-119473, gh-119573, gh-119750, gh-121431, gh-121432, gh-121433, and gh-121435 before making changes; done means the proposed performance improvements preserve exception handling and traversal behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100