python / python/cpython

Importing `readline` leaks memory on macOS

Aberta
#150,536 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

extension-modules OS-mac type-bug
Linguagem predominante
Python
Estrelas
77.2k
Forks
35.9k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

Bug report

Bug description:

In readline.c, setup_readline probes libedit's indexing behaviour by calling add_history twice (with the strings "1" and "2"), replacing one entry via replace_history_entry, and then calling clear_history to clean up.

https://github.com/python/cpython/blob/629da5c914b4407e01c1dc06cbcbd8dce825fef3/Modules/readline.c#L1360-L1397

On macOS, Apple's libedit clear_history does not free the line field (the internally strdup'd string) for all history entries, so one 2-byte allocation escapes. The replaced entry's old value is correctly freed by _py_free_history_entry_lock_held, but clear_history leaks one of the remaining strdup'd probe strings.

This leak (discovered while running unit tests for a Python package with LSan) is reported by Leak Sanitizer as:

Direct leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0x000101f3177c in strdup+0xfc (libclang_rt.asan_osx_dynamic.dylib:arm64+0x4d77c)
    #1 0x0001dbc62ab0  (libedit.3.dylib:arm64e+0x14ab0)
    #2 0x0001dbc63410 in history+0x6e4 (libedit.3.dylib:arm64e+0x15410)
    #3 0x0001dbc5853c in add_history+0x50 (libedit.3.dylib:arm64e+0xa53c)
    #4 0x00010b7b0ad4 in setup_readline readline.c:1373
    #5 0x00010b7b0ad4 in PyInit_readline readline.c:1679
    #6 0x000100ee935c in _PyImport_RunModInitFunc importdl.c:436
    #7 0x000100ee36ac in import_run_extension import.c:2164
    #8 0x000100ee6b90 in _imp_create_dynamic_impl import.c:5503
    #9 0x000100ee6b90 in _imp_create_dynamic import.c.h:488
    #10 0x000100aa5ffc in _PyVectorcall_Call call.c:273
    #11 0x000100e01f68 in _PyEval_EvalFrameDefault generated_cases.c.h:2724
    #12 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #13 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #14 0x000100aaa7c0 in _PyObject_VectorcallTstate pycore_call.h:144
    #15 0x000100aaa7c0 in object_vacall call.c:823
    #16 0x000100aaa29c in PyObject_CallMethodObjArgs call.c:960
    #17 0x000100edf110 in import_find_and_load import.c:4125
    #18 0x000100ede4a8 in PyImport_ImportModuleLevelObject import.c
    #19 0x000100e37658 in _PyEval_ImportNameWithImport ceval.c:3011
    #20 0x000100e37658 in _PyEval_ImportName ceval.c:2990
    #21 0x000100e104ec in _PyEval_EvalFrameDefault generated_cases.c.h:6768
    #22 0x000100af1004 in _PyEval_EvalFrame pycore_ceval.h:122
    #23 0x000100af1004 in gen_send_ex2 genobject.c:280
    #24 0x000100af1004 in gen_send_ex genobject.c:373
    #25 0x000100aed464 in gen_iternext genobject.c:764
    #26 0x000100df48cc in builtin_next bltinmodule.c:1764
    #27 0x000100e14854 in _Py_BuiltinCallFast_StackRef ceval.c:824
    #28 0x000100e14854 in _PyEval_EvalFrameDefault generated_cases.c.h:2420
    #29 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #30 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #31 0x000100aa489c in _PyObject_VectorcallDictTstate call.c:146
    #32 0x000100aa7130 in _PyObject_Call_Prepend call.c:504
    #33 0x000100c292ec in call_method typeobject.c:3095
    #34 0x000100c292ec in slot_tp_call typeobject.c:10913
    #35 0x000100aa4d90 in _PyObject_MakeTpCall call.c:242
    #36 0x000100dfce0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #37 0x000100e1336c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #38 0x000100dfc064 in _PyEval_EvalFrame pycore_ceval.h:122
    #39 0x000100dfc064 in _PyEval_Vector ceval.c:2156
    #40 0x000100aa9208 in _PyObject_VectorcallTstate pycore_call.h:144
    #41 0x000100aa9208 in _PyObject_VectorcallPrepend call.c:877

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 1 allocation(s).

That leak can easily by running the following under an LSan-enabled CPython on macOS.

ASAN_OPTIONS=detect_leaks=1 ./python.exe -c "import readline"

LLM-reviewing this issue surfaces two other issues around that code:

  1. The probing block runs unconditionally even on GNU readline builds, where libedit_history_start and libedit_append_replace_history_offset are never read (every use of those variables is guarded by if (using_libedit_emulation)). This means add_history/clear_history are called unnecessarily on every readline import on Linux.

  2. There is no null check before calling _py_free_history_entry_lock_held(old_entry) on the return value of replace_history_entry, which can return NULL if the index is out of range.

Proposed fix
  • Guard the entire probing block with if (using_libedit_emulation).
  • Replace clear_history inside that block with an explicit remove_history(libedit_history_start) loop that calls _py_free_history_entry_lock_held on each returned entry, bypassing the libedit bug.
  • Keep the unconditional clear_history call outside the block for readline state reset.
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-150537

Guia de contribuição

Abrir o guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Direção de pesquisa

Comece em Modules/readline.c, em setup_readline, e revise o bloco de sondagem do libedit, a limpeza do histórico e a chamada incondicional de clear_history. Reproduza o relatório com ASAN_OPTIONS=detect_leaks=1 ./python.exe -c "import readline" no macOS; considera-se concluído quando o leak desaparecer sem sondagem desnecessária nas builds do GNU readline e a limpeza do histórico relevante continuar segura.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
c, python
Domínio
cli
Tipo de issue
Bug
Dificuldade
3/5
Tempo estimado
1-2 dias
Status de atividade
Estagnada
Clareza
Claramente especificada
Facilidade para iniciantes
25/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.