python / python/cpython

Importing `readline` leaks memory on macOS

未关闭
#150,536 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

extension-modules OS-mac type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

In readline.c, setup_readline probes libedit's indexing behaviour by calling add_history twice (with the strings "1" and "2"), replacing one entry via replace_history_entry, and then calling clear_history to clean up.

https://github.com/python/cpython/blob/629da5c914b4407e01c1dc06cbcbd8dce825fef3/Modules/readline.c#L1360-L1397

On macOS, Apple's libedit clear_history does not free the line field (the internally strdup'd string) for all history entries, so one 2-byte allocation escapes. The replaced entry's old value is correctly freed by _py_free_history_entry_lock_held, but clear_history leaks one of the remaining strdup'd probe strings.

This leak (discovered while running unit tests for a Python package with LSan) is reported by Leak Sanitizer as:

Direct leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0x000101f3177c in strdup+0xfc (libclang_rt.asan_osx_dynamic.dylib:arm64+0x4d77c)
    #1 0x0001dbc62ab0  (libedit.3.dylib:arm64e+0x14ab0)
    #2 0x0001dbc63410 in history+0x6e4 (libedit.3.dylib:arm64e+0x15410)
    #3 0x0001dbc5853c in add_history+0x50 (libedit.3.dylib:arm64e+0xa53c)
    #4 0x00010b7b0ad4 in setup_readline readline.c:1373
    #5 0x00010b7b0ad4 in PyInit_readline readline.c:1679
    #6 0x000100ee935c in _PyImport_RunModInitFunc importdl.c:436
    #7 0x000100ee36ac in import_run_extension import.c:2164
    #8 0x000100ee6b90 in _imp_create_dynamic_impl import.c:5503
    #9 0x000100ee6b90 in _imp_create_dynamic import.c.h:488
    #10 0x000100aa5ffc in _PyVectorcall_Call call.c:273
    #11 0x000100e01f68 in _PyEval_EvalFrameDefault generated_cases.c.h:2724
    #12 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #13 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #14 0x000100aaa7c0 in _PyObject_VectorcallTstate pycore_call.h:144
    #15 0x000100aaa7c0 in object_vacall call.c:823
    #16 0x000100aaa29c in PyObject_CallMethodObjArgs call.c:960
    #17 0x000100edf110 in import_find_and_load import.c:4125
    #18 0x000100ede4a8 in PyImport_ImportModuleLevelObject import.c
    #19 0x000100e37658 in _PyEval_ImportNameWithImport ceval.c:3011
    #20 0x000100e37658 in _PyEval_ImportName ceval.c:2990
    #21 0x000100e104ec in _PyEval_EvalFrameDefault generated_cases.c.h:6768
    #22 0x000100af1004 in _PyEval_EvalFrame pycore_ceval.h:122
    #23 0x000100af1004 in gen_send_ex2 genobject.c:280
    #24 0x000100af1004 in gen_send_ex genobject.c:373
    #25 0x000100aed464 in gen_iternext genobject.c:764
    #26 0x000100df48cc in builtin_next bltinmodule.c:1764
    #27 0x000100e14854 in _Py_BuiltinCallFast_StackRef ceval.c:824
    #28 0x000100e14854 in _PyEval_EvalFrameDefault generated_cases.c.h:2420
    #29 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #30 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #31 0x000100aa489c in _PyObject_VectorcallDictTstate call.c:146
    #32 0x000100aa7130 in _PyObject_Call_Prepend call.c:504
    #33 0x000100c292ec in call_method typeobject.c:3095
    #34 0x000100c292ec in slot_tp_call typeobject.c:10913
    #35 0x000100aa4d90 in _PyObject_MakeTpCall call.c:242
    #36 0x000100dfce0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #37 0x000100e1336c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #38 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #39 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #40 0x000100aa9208 in _PyObject_VectorcallTstate pycore_call.h:144
    #41 0x000100aa9208 in _PyObject_VectorcallPrepend call.c:877

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 1 allocation(s).

That leak can easily by running the following under an LSan-enabled CPython on macOS.

ASAN_OPTIONS=detect_leaks=1 ./python.exe -c "import readline"

LLM-reviewing this issue surfaces two other issues around that code:

  1. The probing block runs unconditionally even on GNU readline builds, where libedit_history_start and libedit_append_replace_history_offset are never read (every use of those variables is guarded by if (using_libedit_emulation)). This means add_history/clear_history are called unnecessarily on every readline import on Linux.

  2. There is no null check before calling _py_free_history_entry_lock_held(old_entry) on the return value of replace_history_entry, which can return NULL if the index is out of range.

Proposed fix
  • Guard the entire probing block with if (using_libedit_emulation).
  • Replace clear_history inside that block with an explicit remove_history(libedit_history_start) loop that calls _py_free_history_entry_lock_held on each returned entry, bypassing the libedit bug.
  • Keep the unconditional clear_history call outside the block for readline state reset.
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-150537

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 Modules/readline.c 中的 setup_readline 开始,检查 libedit 探测代码块、其历史记录清理,以及无条件调用 clear_history 的部分。在 macOS 上使用 ASAN_OPTIONS=detect_leaks=1 ./python.exe -c "import readline" 重现该报告;当 GNU readline 构建中无需进行不必要的探测即可消除泄漏,并且相关的历史记录清理仍然安全时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, python
领域
cli
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。