python / python/cpython

The examples of "c extern" in the "Doc/includes/newtypes" directory of Python 3.11 and later versions need to be optimized, the memory structure of the `__dict__` in the C extern example under `Doc/includes/newtypes` is outdated.

未關閉
#123,951 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

docs
主要語言
Python
星號
77.2k
分支
36k
PR 合併指標
PR 指標待擷取

描述

Documentation

For Python 3.11 and later, refer to:
https://github.com/faster-cpython/ideas/issues/80 (implementation details in CPython source: _PyObject_InitializeDict)
https://www.youtube.com/watch?v=xKk7IXm0XO0
The memory structure of the Python object dict pointer (dict ptr) is optimized:
image

However, in CPython directory Doc/includes/newtypes, the C extern example that involves adding a __dict__, such as:

from custom2 import Custom
class ItemData2(Custom):                                      // c-extern dict-ptr is old-style memory structure
    def __init__(self):
        self.itemid = 0
        self.item = 0

still uses the old dict ptr structure (prior to Python 3.10).
image

This involves two issues:

  1. The dict ptr structure uses the previous type (as stated above)
  2. The dict ptr memory optimization in Python 3.11 and later does not work in the C extern example:

The memory optimization in _PyObject_InitializeDict->init_inline_values is not utilized:

static int
init_inline_values(PyObject *obj, PyTypeObject *tp)
{
    assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE);
    // assert(type->tp_dictoffset > 0);  -- TODO: Update this assert.
    assert(tp->tp_flags & Py_TPFLAGS_MANAGED_DICT);
    PyDictKeysObject *keys = CACHED_KEYS(tp);
    assert(keys != NULL);
    if (keys->dk_usable > 1) {
        keys->dk_usable--; // Key optimization for memory when creating many objects with __dict__
    }
}

When objects are created in large quantities, the memory per single object can differ by nearly 200 ~ 300bytes 。

detail
memory of ItemData2 = 392 bytes
typedef struct {
    PyObject_HEAD
} CustomObject;   # remove custom2.c CustomObject field: first, last, number

"The memory size allocated for new_values as PyDictValues is determined by arg size
image
arg size default value is 30(when the __dict__ contains fewer than 30 members):
image

When arg size is 30, the memory size of PyDictValues is 272 bytes:"

image

total size:

56(object ptr basic) + 64(dict ptr) + 0(cache key ptr, shared memory) + 272(values ptr) = 392 bytes
memory of ItemData = 104 bytes

when the __dict__ contains 2 memebers:

memory of values ptr = prefix_size + size * sizeof(PyObject*) = prefix_size + (dk_entries + dk_usable) * sizeof(PyObject*) 
= 32 + (2 + 0) * 8 = 48 bytes
56(object basic memory) + 0(dict ptr) + 0(cache key, shared) + 48(value ptr)= 104 bytes 

I try to fix it, like:

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CustomObject *self;

    // 直接调用object.tp_new, 注意不要把 args 直接传PyBaseObject_Type.tp_new,`__init__`带参数时会报错
    static PyObject* intern_tuple0 = NULL; 
    if (intern_tuple0 == NULL) 
        intern_tuple0 = PyTuple_New(0);
    self = PyBaseObject_Type.tp_new(type, intern_tuple0, NULL);

    //self = (CustomObject *) type->tp_alloc(type, 0);

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

從 Doc/includes/newtypes 開始,檢查新增 dict 的 C extern 範例,然後將它們的配置行為與 Python 3.11+ 和 _PyObject_InitializeDict 進行比較。使用所參照的 faster-cpython issue 和 CPython 實作細節來判定目前的結構;完成的標準是這些範例反映受支援的配置並使用預期的記憶體最佳化。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
c, python
領域
documentation
Issue 類型
文件
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。