python / python/cpython

The examples of "c extern" in the "Doc/includes/newtypes" directory of Python 3.11 and later versions need to be optimized, the memory structure of the `__dict__` in the C extern example under `Doc/includes/newtypes` is outdated.

Abierto
#123,951 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

docs
Lenguaje dominante
Python
Estrellas
77.2k
Forks
35.9k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Documentation

For Python 3.11 and later, refer to:
https://github.com/faster-cpython/ideas/issues/80 (implementation details in CPython source: _PyObject_InitializeDict)
https://www.youtube.com/watch?v=xKk7IXm0XO0
The memory structure of the Python object dict pointer (dict ptr) is optimized:
image

However, in CPython directory Doc/includes/newtypes, the C extern example that involves adding a __dict__, such as:

from custom2 import Custom
class ItemData2(Custom):                                      // c-extern dict-ptr is old-style memory structure
    def __init__(self):
        self.itemid = 0
        self.item = 0

still uses the old dict ptr structure (prior to Python 3.10).
image

This involves two issues:

  1. The dict ptr structure uses the previous type (as stated above)
  2. The dict ptr memory optimization in Python 3.11 and later does not work in the C extern example:

The memory optimization in _PyObject_InitializeDict->init_inline_values is not utilized:

static int
init_inline_values(PyObject *obj, PyTypeObject *tp)
{
    assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE);
    // assert(type->tp_dictoffset > 0);  -- TODO: Update this assert.
    assert(tp->tp_flags & Py_TPFLAGS_MANAGED_DICT);
    PyDictKeysObject *keys = CACHED_KEYS(tp);
    assert(keys != NULL);
    if (keys->dk_usable > 1) {
        keys->dk_usable--; // Key optimization for memory when creating many objects with __dict__
    }
}

When objects are created in large quantities, the memory per single object can differ by nearly 200 ~ 300bytes 。

detail
memory of ItemData2 = 392 bytes
typedef struct {
    PyObject_HEAD
} CustomObject;   # remove custom2.c CustomObject field: first, last, number

"The memory size allocated for new_values as PyDictValues is determined by arg size
image
arg size default value is 30(when the __dict__ contains fewer than 30 members):
image

When arg size is 30, the memory size of PyDictValues is 272 bytes:"

image

total size:

56(object ptr basic) + 64(dict ptr) + 0(cache key ptr, shared memory) + 272(values ptr) = 392 bytes
memory of ItemData = 104 bytes

when the __dict__ contains 2 memebers:

memory of values ptr = prefix_size + size * sizeof(PyObject*) = prefix_size + (dk_entries + dk_usable) * sizeof(PyObject*) 
= 32 + (2 + 0) * 8 = 48 bytes
56(object basic memory) + 0(dict ptr) + 0(cache key, shared) + 48(value ptr)= 104 bytes 

I try to fix it, like:

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CustomObject *self;

    // 直接调用object.tp_new, 注意不要把 args 直接传PyBaseObject_Type.tp_new,`__init__`带参数时会报错
    static PyObject* intern_tuple0 = NULL; 
    if (intern_tuple0 == NULL) 
        intern_tuple0 = PyTuple_New(0);
    self = PyBaseObject_Type.tp_new(type, intern_tuple0, NULL);

    //self = (CustomObject *) type->tp_alloc(type, 0);

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Comienza en Doc/includes/newtypes e inspecciona los ejemplos de C extern que añaden un dict; después compara su comportamiento de asignación con Python 3.11+ y _PyObject_InitializeDict. Usa el faster-cpython issue referenciado y los detalles de implementación de CPython para determinar la estructura actual; el trabajo está terminado cuando los ejemplos reflejan el layout compatible y utilizan la optimización de memoria prevista.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
c, python
Área
documentation
Tipo de issue
Documentación
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.