python / python/cpython

Improve `object.__reduce_ex__` performance up to 20%

未关闭
#148,269 4 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

interpreter-core performance type-feature
主要语言
Python
星标
77.2k
派生
35.9k
PR 指标待抓取

描述

Feature or enhancement

Context

Currently, when object.__reduce_ex__ is called, it might forward the call to __reduce__.

For this to happen the following must be true:

  1. __reduce__ must be defined as class attribute and must not be object.__reduce__
  2. __reduce__ must be defined as instance attribute

Roughly the following logic is ran:

class object:
    def __reduce_ex__(self, protocol: int) -> tuple:
        reduce = getattr(self, "__reduce__", None)  # <-- might not be necessary
        if reduce is not None:
            if getattr(type(self), "__reduce__", None) is not object.__reduce__:  # <-- real descriminator
                return reduce()
            # `reduce` above is thrown away, but we paid upfront cost to look it up
        return self._common_reduce(protocol)
Actual code (toggle visibility)

https://github.com/python/cpython/blob/eab7dbda3b7502f0a952901a80fb5e628ccd7a28/Objects/typeobject.c#L8245-L8277

Looking at this it's evident, that we always need to check the first condition, but might not need to check the second one.

Proposal

  1. Lookup instance attribute only after class-level override is confirmed
  2. Replace PyObject_GetAttr to _Py_LookupRef — just a little speedup
def _PyType_Lookup(cls, name):
    for base in cls.__mro__:
        if name in base.__dict__:
            return base.__dict__[name]
    return None

class object:
    def __reduce_ex__(self, protocol: int) -> tuple:
        if _PyType_Lookup(type(self), "__reduce__") is not object.__reduce__:
            reduce = getattr(self, "__reduce__", None)
            if reduce is not None:
                return reduce()
        return self._common_reduce(protocol)

Why not

Proposed change would make the following code behave differently (this is the only case when behavior differs I've been able to come up with):

class X:
    def __getattribute__(self, name):
        if name == "__reduce__":
            raise RuntimeError("Boom!")
        return object.__getattribute__(self, name)

x = X()
x.__reduce_ex__(5)  # current implementation will raise, proposed will not

Descriptors behavior is not affected by the change since they will always be defined as class attributes.

Potential speedup

Benchmark current patched
default_reduce_ex 297 ns 250 ns: 1.19x faster
slots_default_reduce_ex 283 ns 239 ns: 1.19x faster
instance_shadow_reduce_ex 278 ns 250 ns: 1.11x faster
small_dataclass_reduce_ex 299 ns 252 ns: 1.19x faster
class_override_reduce_ex 148 ns 137 ns: 1.08x faster
class_override_getattribute_reduce_ex 362 ns 348 ns: 1.04x faster
pickle_dumps_default 1.44 us 1.29 us: 1.12x faster
pickle_dumps_class_override 1.19 us 1.15 us: 1.03x faster
pickle_small_dataclass 1.50 us 1.45 us: 1.04x faster
Geometric mean (ref) 1.10x faster
Benchmark (toggle visibility) 
"""
pyperf benchmarks for object.__reduce_ex__ behavior relevant to a CPython patch
that changes the lookup order from:

    1. self.__reduce__
    2. type(self).__reduce__

to:

    1. type(self).__reduce__
    2. self.__reduce__ only if the class actually overrides it

Usage:
    ./python bench_reduce_ex.py
    ./python bench_reduce_ex.py -o patched.json

Compare:
    python -m pyperf compare_to baseline.json patched.json
"""

import pickle
from dataclasses import dataclass
from functools import partial

import pyperf


class Default:
    pass


default_obj = Default()


class SlotsDefault:
    __slots__ = ()


slots_default_obj = SlotsDefault()


class DefaultWithInstanceReduce:
    pass


instance_shadow_obj = DefaultWithInstanceReduce()
instance_shadow_obj.__reduce__ = lambda: (DefaultWithInstanceReduce, ())


class ClassOverride:
    def __reduce__(self):
        return ClassOverride, ()


class_override_obj = ClassOverride()


class ClassOverrideWithGetattribute:
    def __getattribute__(self, name):
        return object.__getattribute__(self, name)

    def __reduce__(self):
        return ClassOverrideWithGetattribute, ()


class_override_getattribute_obj = ClassOverrideWithGetattribute()


@dataclass
class Data:
    x: int
    y: str


small_dataclass = Data(42, "foo")


def main():
    runner = pyperf.Runner()

    runner.bench_func(
        "default_reduce_ex",
        partial(default_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "slots_default_reduce_ex",
        partial(slots_default_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "instance_shadow_reduce_ex",
        partial(instance_shadow_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "small_dataclass_reduce_ex",
        partial(small_dataclass.__reduce_ex__, 4),
    )
    runner.bench_func(
        "class_override_reduce_ex",
        partial(class_override_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "class_override_getattribute_reduce_ex",
        partial(class_override_getattribute_obj.__reduce_ex__, 4),
    )

    runner.bench_func(
        "pickle_dumps_default",
        partial(pickle.dumps, default_obj),
    )
    runner.bench_func(
        "pickle_dumps_slots_default",
        partial(pickle.dumps, slots_default_obj),
    )
    runner.bench_func(
        "pickle_dumps_class_override",
        partial(pickle.dumps, class_override_obj),
    )
    runner.bench_func(
        "pickle_small_dataclass",
        partial(pickle.dumps, small_dataclass),
    )


if __name__ == "__main__":
    main()
Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-148281

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

调研方向

从 Objects/typeobject.c 中 object.reduce_ex 附近的链接行开始，然后查看提议的查找顺序变更及其行为示例。使用内置的 pyperf 基准测试 bench_reduce_ex.py，比较列出的 reduce_ex 和 pickle 用例；当优化在保持所述行为的同时提升报告的性能时，即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, python
领域
compilers, performance
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。