python / python/cpython

[C-API Docs] Clarify Constraints on the Arena Allocator

未關閉
#101,655 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

3.11 3.12 3.13 docs topic-C-API
主要語言
Python
星號
77.2k
分支
36k
PR 合併指標
PR 指標待擷取

描述

The "arena" allocator is used by obmalloc, as well as the code for allocating frames. It is exposed by the following public C-API:

  • PyObject_GetArenaAllocator()
  • PyObject_SetArenaAllocator()

(It is also exposed/used internally as _PyObject_VirtualAlloc() and _PyObject_VirtualFree(). The implementation is exposed internally as _PyMem_ArenaAlloc() and _PyMem_ArenaFree().)

First of all, the docs for PyObject_GetArenaAllocator() are not clear at all about what the function should be used for. I expect it is only meant for use in combination with PyObject_SetArenaAllocator() (e.g. to wrap the old one; to check what's set). That should be clarified.

Second, the C-API docs make it clear that the two public functions may be called before the runtime is initialized. However, the following should be clarified, as there are no documented restrictions AFAICS:

  • may PyObject_SetArenaAllocator() be called after runtime init?
    • status quo: presumably not, since obmalloc would break (similar to the restrictions on PyMem_SetAllocator())
    • internally: we do not ever change the arena allocator
    • recommendation: explicitly document that it must not be called after runtime init (a la PyMem_SetAllocator())
  • may the allocator returned by PyObject_GetArenaAllocator() be used directly?
    • status quo: possibly yes, since people may be doing so at this point (unlikely though)
    • internally: we use it directly (but that's kind of the point)
    • recommendation: explicitly state that it should not be used directly (point to object allocator API instead)
  • ...before runtime init?
    • (relevant only if direct use is okay)
    • status quo: possibly yes, since people may be doing so at this point (unlikely though)
    • internally: we do not use it before runtime init
    • recommendation: at the least document that it shouldn't be used after runtime init
    • better: state that it's only allowed when PyObject_SetArenaAllocator() is allowed (i.e. only before init)
  • must PyObject_GetArenaAllocator() be called with the GIL held after runtime init?
    • (relevant only if post-init PyObject_SetArenaAllocator() is okay)
    • status quo: possibly not, since people haven't needed to before (unlikely though)
    • internally: we always hold the GIL
    • recommendation: recommend (or require) that the GIL be held
  • must the allocator passed to PyObject_SetArenaAllocator() be thread-safe (and reentrant)?
    • status quo: if the allocator may be used directly then possibly yes (since it seems it can be used before runtime init, as well as later without the GIL held); otherwise no (since the PyMem "object" allocator is documented as protected by the GIL)
    • internally: the default arena allocator is thread-safe (and reentrant)
    • recommendation: recommend that the arena allocator be thread-safe (just like the PyMem "raw" allocator)
    • better: explicitly require it (so we can later drop the restriction on the PyMem "mem" and "object" allocators)

If we do specify tighter restrictions than were already documented then we'll need to consider who might be using the public functions in unexpected ways. However, I'd be surprised if there were more than a few (if any) using them at all.


Regardless, ISTM it would be better to deprecate the two public functions and replace them with a new PyConfig field. That would align a lot better with the intention. If there is a need to see if the currently set allocator matches some expected one, then it would be better to have an explicit function for that (e.g. PyObject_IsArenaAllocator()) rather than exposing PyObject_GetArenaAllocator() for that purpose.

The same goes for PyMem_SetAllocator(), where we'd have an additional PyMem_SetAllocatorWrapper() for the after-init case.

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

先從連結的 C-API 文件開始,查看 PyObject_GetArenaAllocator()、PyObject_SetArenaAllocator() 以及 Python 初始化前的指引,接著比較 PyMem_SetAllocator 的限制。確定並記錄初始化、GIL、直接使用和執行緒安全方面允許的約束;另外評估是否應指定棄用以及使用 PyConfig 的替代方案。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
c, python
領域
api, documentation
Issue 類型
文件
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。