python / python/cpython

The examples of "c extern" in the "Doc/includes/newtypes" directory of Python 3.11 and later versions need to be optimized, the memory structure of the `__dict__` in the C extern example under `Doc/includes/newtypes` is outdated.

未关闭
#123,951 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

docs
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Documentation

For Python 3.11 and later, refer to:
https://github.com/faster-cpython/ideas/issues/80 (implementation details in CPython source: _PyObject_InitializeDict)
https://www.youtube.com/watch?v=xKk7IXm0XO0
The memory structure of the Python object dict pointer (dict ptr) is optimized:
image

However, in CPython directory Doc/includes/newtypes, the C extern example that involves adding a __dict__, such as:

from custom2 import Custom
class ItemData2(Custom):                                      // c-extern dict-ptr is old-style memory structure
    def __init__(self):
        self.itemid = 0
        self.item = 0

still uses the old dict ptr structure (prior to Python 3.10).
image

This involves two issues:

  1. The dict ptr structure uses the previous type (as stated above)
  2. The dict ptr memory optimization in Python 3.11 and later does not work in the C extern example:

The memory optimization in _PyObject_InitializeDict->init_inline_values is not utilized:

static int
init_inline_values(PyObject *obj, PyTypeObject *tp)
{
    assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE);
    // assert(type->tp_dictoffset > 0);  -- TODO: Update this assert.
    assert(tp->tp_flags & Py_TPFLAGS_MANAGED_DICT);
    PyDictKeysObject *keys = CACHED_KEYS(tp);
    assert(keys != NULL);
    if (keys->dk_usable > 1) {
        keys->dk_usable--; // Key optimization for memory when creating many objects with __dict__
    }
}

When objects are created in large quantities, the memory per single object can differ by nearly 200 ~ 300bytes 。

detail
memory of ItemData2 = 392 bytes
typedef struct {
    PyObject_HEAD
} CustomObject;   # remove custom2.c CustomObject field: first, last, number

"The memory size allocated for new_values as PyDictValues is determined by arg size
image
arg size default value is 30(when the __dict__ contains fewer than 30 members):
image

When arg size is 30, the memory size of PyDictValues is 272 bytes:"

image

total size:

56(object ptr basic) + 64(dict ptr) + 0(cache key ptr, shared memory) + 272(values ptr) = 392 bytes
memory of ItemData = 104 bytes

when the __dict__ contains 2 memebers:

memory of values ptr = prefix_size + size * sizeof(PyObject*) = prefix_size + (dk_entries + dk_usable) * sizeof(PyObject*) 
= 32 + (2 + 0) * 8 = 48 bytes
56(object basic memory) + 0(dict ptr) + 0(cache key, shared) + 48(value ptr)= 104 bytes 

I try to fix it, like:

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CustomObject *self;

    // 直接调用object.tp_new, 注意不要把 args 直接传PyBaseObject_Type.tp_new,`__init__`带参数时会报错
    static PyObject* intern_tuple0 = NULL; 
    if (intern_tuple0 == NULL) 
        intern_tuple0 = PyTuple_New(0);
    self = PyBaseObject_Type.tp_new(type, intern_tuple0, NULL);

    //self = (CustomObject *) type->tp_alloc(type, 0);

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 Doc/includes/newtypes 开始,检查添加 dict 的 C extern 示例,然后将它们的分配行为与 Python 3.11+ 和 _PyObject_InitializeDict 进行比较。使用所引用的 faster-cpython issue 和 CPython 实现细节来确定当前结构;完成的标准是这些示例反映受支持的布局并使用预期的内存优化。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, python
领域
documentation
Issue 类型
文档
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。