Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
SimpleHTTPRequestHandler.list_directory() calls os.listdir() and then, for every entry, os.path.isdir() (a stat) and os.path.islink() (an lstat) — two stat-family syscalls per entry. This is wasted work on any filesystem and dominates listing time for large directories; on network filesystems like NFS, where each call is a round-trip, it becomes severe.
os.scandir() returns the entry type from the directory read itself (POSIX d_type / NFS READDIRPLUS), eliminating the per-entry stats in the common case. CPython already did this migration for os.walk(), glob, and pathlib.Path.iterdir() (gh-117727); http.server was missed.
Benchmark
Directory with 1000 files + 1000 dirs (plus a few symlinks):
- stat-family syscalls (
strace): 4088 → 88 (the 88 is constant interpreter startup; the per-entry loop drops from ~2 syscalls to ~0) - local filesystem wall-clock: ~10× faster
- emulating NFS by injecting per-
statlatency: the listing goes from seconds to ~2 ms
Worst case — a mount that returns DT_UNKNOWN — falls back to one cached lstat per entry, which is still fewer calls than today and never worse.
The change is behavior-preserving: DirEntry.is_dir()/is_symlink() match os.path.isdir/os.path.islink semantics (follow-symlinks behavior and return-False-on-error), verified across real dirs/files, symlink-to-dir, symlink-to-file, and broken symlinks. The existing test_httpservers suite passes unchanged.
I have a patch ready and will open a PR.
This issue was prepared with AI assistance (Claude Code); the analysis and benchmarks were reviewed by me.
Linked PRs
- gh-151789
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at SimpleHTTPRequestHandler.list_directory() and inspect the existing test_httpservers suite. Compare the directory-entry checks with os.scandir() behavior, including symlinks and fallback cases, then run the HTTP server tests; done means behavior remains unchanged while per-entry stat calls are reduced. A linked pull request already covers this work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance, web-dev
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100