python / python/cpython

Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir()

未关闭
#151,788 12 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

performance stdlib triaged type-feature
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

SimpleHTTPRequestHandler.list_directory() calls os.listdir() and then, for every entry, os.path.isdir() (a stat) and os.path.islink() (an lstat) — two stat-family syscalls per entry. This is wasted work on any filesystem and dominates listing time for large directories; on network filesystems like NFS, where each call is a round-trip, it becomes severe.

os.scandir() returns the entry type from the directory read itself (POSIX d_type / NFS READDIRPLUS), eliminating the per-entry stats in the common case. CPython already did this migration for os.walk(), glob, and pathlib.Path.iterdir() (gh-117727); http.server was missed.

Benchmark

Directory with 1000 files + 1000 dirs (plus a few symlinks):

  • stat-family syscalls (strace): 4088 → 88 (the 88 is constant interpreter startup; the per-entry loop drops from ~2 syscalls to ~0)
  • local filesystem wall-clock: ~10× faster
  • emulating NFS by injecting per-stat latency: the listing goes from seconds to ~2 ms

Worst case — a mount that returns DT_UNKNOWN — falls back to one cached lstat per entry, which is still fewer calls than today and never worse.

The change is behavior-preserving: DirEntry.is_dir()/is_symlink() match os.path.isdir/os.path.islink semantics (follow-symlinks behavior and return-False-on-error), verified across real dirs/files, symlink-to-dir, symlink-to-file, and broken symlinks. The existing test_httpservers suite passes unchanged.

I have a patch ready and will open a PR.


This issue was prepared with AI assistance (Claude Code); the analysis and benchmarks were reviewed by me.

Linked PRs
  • gh-151789

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 SimpleHTTPRequestHandler.list_directory() 开始,检查现有的 test_httpservers 测试套件。将目录条目检查与 os.scandir() 的行为进行比较,包括符号链接和回退情况,然后运行 HTTP 服务器测试;当行为保持不变且每个条目的 stat 调用次数减少时,即表示完成。一个已链接的 pull request 已经涵盖了这项工作。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend, performance, web-dev
Issue 类型
重构
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。