python / python/cpython

Lock contention inside `_PyType_LookupRef`

Open
#132,380 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

interpreter-core performance topic-free-threading type-feature
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Feature or enhancement

Proposal:

As part of adding free-threading support to LibCST, we noticed there is a lot of lock contention on TYPE_LOCK inside the _PyType_LookupRef function. In the LibCST, the common "visitor" pattern is used. For example, in the _visitors.py module there is the code:

visit_func = getattr(self, f"visit_{type(node).__name__}", None)

The second argument to getattr() is a non-interned string and it causes the cached and lock-free path of _PyType_LookupRef() never to be taken. Instead, the TYPE_LOCK mutex is acquired on each lookup. This obviously scales very badly if there are multiple threads looking up class methods using this pattern.

Testing was done with Python 3.13 but I believe the same issue exists with 3.14.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-132381
  • gh-131174
  • gh-132651
  • gh-132652
  • gh-133669
  • gh-135112

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading _PyType_LookupRef and the TYPE_LOCK path in CPython, then inspect the visitor pattern in LibCST’s _visitors.py. Compare the behavior on Python 3.13 and 3.14 and review the linked PRs; done should include an agreed change that addresses the lock contention and evidence that the lookup path scales better.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.