python / python/cpython

Improve `object.__reduce_ex__` performance up to 20%

オープン
#148,269 コメント 4 件 リアクション 2 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

interpreter-core performance type-feature
主要言語
Python
スター
77.2k
フォーク
35.9k
PR 指標を取得中

説明

Feature or enhancement

Context

Currently, when object.__reduce_ex__ is called, it might forward the call to __reduce__.

For this to happen the following must be true:

  1. __reduce__ must be defined as class attribute and must not be object.__reduce__
  2. __reduce__ must be defined as instance attribute

Roughly the following logic is ran:

class object:
    def __reduce_ex__(self, protocol: int) -> tuple:
        reduce = getattr(self, "__reduce__", None)  # <-- might not be necessary
        if reduce is not None:
            if getattr(type(self), "__reduce__", None) is not object.__reduce__:  # <-- real descriminator
                return reduce()
            # `reduce` above is thrown away, but we paid upfront cost to look it up
        return self._common_reduce(protocol)
Actual code (toggle visibility)

https://github.com/python/cpython/blob/eab7dbda3b7502f0a952901a80fb5e628ccd7a28/Objects/typeobject.c#L8245-L8277

Looking at this it's evident, that we always need to check the first condition, but might not need to check the second one.

Proposal

  1. Lookup instance attribute only after class-level override is confirmed
  2. Replace PyObject_GetAttr to _Py_LookupRef — just a little speedup
def _PyType_Lookup(cls, name):
    for base in cls.__mro__:
        if name in base.__dict__:
            return base.__dict__[name]
    return None

class object:
    def __reduce_ex__(self, protocol: int) -> tuple:
        if _PyType_Lookup(type(self), "__reduce__") is not object.__reduce__:
            reduce = getattr(self, "__reduce__", None)
            if reduce is not None:
                return reduce()
        return self._common_reduce(protocol)

Why not

Proposed change would make the following code behave differently (this is the only case when behavior differs I've been able to come up with):

class X:
    def __getattribute__(self, name):
        if name == "__reduce__":
            raise RuntimeError("Boom!")
        return object.__getattribute__(self, name)

x = X()
x.__reduce_ex__(5)  # current implementation will raise, proposed will not

Descriptors behavior is not affected by the change since they will always be defined as class attributes.

Potential speedup

Benchmark current patched
default_reduce_ex 297 ns 250 ns: 1.19x faster
slots_default_reduce_ex 283 ns 239 ns: 1.19x faster
instance_shadow_reduce_ex 278 ns 250 ns: 1.11x faster
small_dataclass_reduce_ex 299 ns 252 ns: 1.19x faster
class_override_reduce_ex 148 ns 137 ns: 1.08x faster
class_override_getattribute_reduce_ex 362 ns 348 ns: 1.04x faster
pickle_dumps_default 1.44 us 1.29 us: 1.12x faster
pickle_dumps_class_override 1.19 us 1.15 us: 1.03x faster
pickle_small_dataclass 1.50 us 1.45 us: 1.04x faster
Geometric mean (ref) 1.10x faster
Benchmark (toggle visibility) 
"""
pyperf benchmarks for object.__reduce_ex__ behavior relevant to a CPython patch
that changes the lookup order from:

    1. self.__reduce__
    2. type(self).__reduce__

to:

    1. type(self).__reduce__
    2. self.__reduce__ only if the class actually overrides it

Usage:
    ./python bench_reduce_ex.py
    ./python bench_reduce_ex.py -o patched.json

Compare:
    python -m pyperf compare_to baseline.json patched.json
"""

import pickle
from dataclasses import dataclass
from functools import partial

import pyperf


class Default:
    pass


default_obj = Default()


class SlotsDefault:
    __slots__ = ()


slots_default_obj = SlotsDefault()


class DefaultWithInstanceReduce:
    pass


instance_shadow_obj = DefaultWithInstanceReduce()
instance_shadow_obj.__reduce__ = lambda: (DefaultWithInstanceReduce, ())


class ClassOverride:
    def __reduce__(self):
        return ClassOverride, ()


class_override_obj = ClassOverride()


class ClassOverrideWithGetattribute:
    def __getattribute__(self, name):
        return object.__getattribute__(self, name)

    def __reduce__(self):
        return ClassOverrideWithGetattribute, ()


class_override_getattribute_obj = ClassOverrideWithGetattribute()


@dataclass
class Data:
    x: int
    y: str


small_dataclass = Data(42, "foo")


def main():
    runner = pyperf.Runner()

    runner.bench_func(
        "default_reduce_ex",
        partial(default_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "slots_default_reduce_ex",
        partial(slots_default_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "instance_shadow_reduce_ex",
        partial(instance_shadow_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "small_dataclass_reduce_ex",
        partial(small_dataclass.__reduce_ex__, 4),
    )
    runner.bench_func(
        "class_override_reduce_ex",
        partial(class_override_obj.__reduce_ex__, 4),
    )
    runner.bench_func(
        "class_override_getattribute_reduce_ex",
        partial(class_override_getattribute_obj.__reduce_ex__, 4),
    )

    runner.bench_func(
        "pickle_dumps_default",
        partial(pickle.dumps, default_obj),
    )
    runner.bench_func(
        "pickle_dumps_slots_default",
        partial(pickle.dumps, slots_default_obj),
    )
    runner.bench_func(
        "pickle_dumps_class_override",
        partial(pickle.dumps, class_override_obj),
    )
    runner.bench_func(
        "pickle_small_dataclass",
        partial(pickle.dumps, small_dataclass),
    )


if __name__ == "__main__":
    main()
Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-148281

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

Objects/typeobject.c の object.reduce_ex 付近にあるリンク先の行から始め、続いて提案されている検索順序の変更と、その動作例を確認してください。組み込まれている pyperf ベンチマーク bench_reduce_ex.py を使って、一覧にある reduce_ex と pickle のケースを比較してください。最適化によって記載された動作が維持され、同時に報告されたパフォーマンスが向上すれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
c, python
領域
compilers, performance
issue の種類
機能追加
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。