python / python/cpython

Types with `mro()` unconditionally disables type attribute cache, causing performance regression for compatible MRO implementations, such as Zope

オープン
#156,096 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

type-bug
主要言語
Python
スター
77.2k
フォーク
35.9k
PR マージ指標
PR 指標を取得中

説明

Bug report

Bug description:

Since https://github.com/python/cpython/issues/127773 / https://github.com/python/cpython/pull/127924, any type whose metaclass defines a custom mro() method has its type attribute cache permanently disabled. This was made initially to fix a problem with metaclasses defining mro() to inject new bases that are not in the class bases.

This affects Zope's ExtensionClass, used as base class for most classes the Zope ecosystem. This package has 5000 daily downloads on pypi ( https://pypistats.org/packages/extensionclass ). ExtensionClass defines a custom mro(), but only to reorganizes bases that are already bases of the class, which is different from the scenario from the issue and it looks like a case where the cache could still be used.

Compared to python3.11, this seems to causes a ~3x performance regression for attribute lookups on all zope classes:

$ uv run --python=3.11 --with Zope python -m timeit -s 'import OFS.Folder; f = OFS.Folder.Folder("f")' 'f.getId'
2000000 loops, best of 5: 116 nsec per loop
$ uv run --python=3.14 --with Zope python -m timeit -s 'import OFS.Folder; f = OFS.Folder.Folder("f")' 'f.getId'
500000 loops, best of 5: 399 nsec per loop

it's hard to tell how much this affect applications, this is just a micro benchmark, but we are also observing that running ERP5 test suites (a complex application based on zope) is slower on python3.13 than it was on python3.11.

Analysis

In type_mro_modified(), the check has_custom_mro(type) unconditionally jumps to clear:

https://github.com/python/cpython/blob/af49df919dafc3767ae956767dce0482f9cd6d4e/Objects/typeobject.c#L1261-L1262

which sets tp_versions_used = _Py_ATTR_CACHE_UNUSED, whenever the metaclass overrides mro():

https://github.com/python/cpython/blob/af49df919dafc3767ae956767dce0482f9cd6d4e/Objects/typeobject.c#L1279-L1283

This was introduced to fix https://github.com/python/cpython/issues/127773, where a custom mro() that injects a class not in tp_bases (e.g. Base) causes stale cache entries because PyType_Modified() only propagates through tp_subclasses, not through MRO-only relationships.

Although these are not much relevant here, ExtensionClass's mro is implemented in C as _ExtensionClass.c:569-626 and also has an equivalent pure-python implementation in ExtensionClass.__init__.py:181-200.

Suggested fix

Check whether the custom MRO actually contains non-base entries before disabling the cache. This requires an is_superclass() helper that walks tp_bases (not tp_mro) to determine reachability:

// Return true if `super` is reachable from `sub` through `tp_bases`,
// as opposed to a type merely injected into the MRO by a custom `mro()`
// implementation.  Only entries reachable through `tp_bases` are covered
// by the version-tag invalidation that _PyType_Modified_Unlocked()
// propagates through `tp_subclasses`.
static int
is_superclass(PyTypeObject *super, PyTypeObject *sub)
{
    PyObject *bases = lookup_tp_bases(sub);
    if (bases == NULL) {
        return 0;
    }
    Py_ssize_t n = PyTuple_GET_SIZE(bases);
    for (Py_ssize_t i = 0; i < n; i++) {
        PyTypeObject *base = _PyType_CAST(PyTuple_GET_ITEM(bases, i));
        if (base == super || is_superclass(super, base)) {
            return 1;
        }
    }
    return 0;
}

Then in type_mro_modified, replace the unconditional goto clear for custom MROs:

if (!Py_IS_TYPE(type, &PyType_Type) && has_custom_mro(type)) {
    // Only disable cache if the custom mro() injected an entry that
    // is not reachable through tp_bases;
    PyObject *mro = lookup_tp_mro(type);
    if (mro == NULL) {
        goto clear;
    }
    Py_ssize_t mro_size = PyTuple_GET_SIZE(mro);
    for (Py_ssize_t m = 0; m < mro_size; m++) {
        PyTypeObject *entry = _PyType_CAST(PyTuple_GET_ITEM(mro, m));
        if (entry != type && !is_superclass(entry, type)) {
            goto clear;
        }
    }
}

I'm not at all familiar with this, but it seems to me that this would preserve the correctness of https://github.com/python/cpython/pull/127924 for types that do inject non-base entries into their MRO (because cache would still be disabled for those), while restoring cache performance for compatible custom MRO implementations like Zope's ExtensionClass.

If this approach makes sense, I already have a draft commit implementing this: https://github.com/perrinjerome/cpython/commit/89af3385eb7e930466dcaef97b2d7b9286eb755e and I would be happy to clean it up and make a pull request.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

Objects/typeobject.c の type_mro_modified() から始め、周辺のキャッシュ無効化ロジックを読みます。次に、リンクされているカスタム MRO の動作を、提案されているドラフトコミットと比較します。tp_bases の外部から挿入された MRO エントリに対するキャッシュ無効化を維持しつつ、互換性のあるカスタム MRO に対するキャッシュを保持し、両方のケースに対するリグレッションカバレッジを追加できれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
c, python
領域
backend, performance
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。