python / python/cpython

Document that bf_getbuffer must prevent GC (only under freethreading?) to write to its `view` argument

Ouverte
#130,409 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

docs topic-free-threading
Langage dominant
Python
Étoiles
77.2k
Forks
35.9k
Métriques de merge des PR
Métriques de PR en attente

Description

https://docs.python.org/3/c-api/typeobj.html#c.PyBufferProcs.bf_getbuffer

does not mention any requirement for the callee to hold the GIL or otherwise prevent Python GC to write to the view argument. One might think this is safe, because one has no particular reason to think that the view object is shared between threads and that the bf_getbuffer implementation has exclusive access to it.

However, the view object can be allocated from GC memory, e.g., in https://github.com/python/cpython/blob/4f07fd59cbd2656ae31e164098b8d10c5b7e51f0/Objects/memoryobject.c#L72
and this means that we may see concurrent GC traversals if we do not prevent GC (e.g., by holding the GIL).

Under free-threading mode, I saw the following race in the JAX test suite which I believe stems from this problem:

WARNING: ThreadSanitizer: data race (pid=268192)
  Read of size 8 at 0x7fffc2221a38 by thread T69 (mutexes: read M0):
    #0 mbuf_traverse /__w/jax/jax/cpython/Objects/memoryobject.c:134:5 (python3.13+0x27f190) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #1 update_refs /__w/jax/jax/cpython/Python/gc_free_threading.c:441:5 (python3.13+0x44854a) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #2 _mi_heap_area_visit_blocks /__w/jax/jax/cpython/Objects/mimalloc/heap.c:630:14 (python3.13+0x2a9fe4) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #3 mi_heap_area_visitor /__w/jax/jax/cpython/Objects/mimalloc/heap.c:681:12 (python3.13+0x2aa547) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #4 mi_heap_visit_areas_page /__w/jax/jax/cpython/Objects/mimalloc/heap.c:661:10 (python3.13+0x2aa547)
    #5 mi_heap_visit_pages /__w/jax/jax/cpython/Objects/mimalloc/heap.c:46:12 (python3.13+0x2aa547)
    #6 mi_heap_visit_areas /__w/jax/jax/cpython/Objects/mimalloc/heap.c:667:10 (python3.13+0x2aa547)
    #7 mi_heap_visit_blocks /__w/jax/jax/cpython/Objects/mimalloc/heap.c:692:10 (python3.13+0x2aa547)
    #8 gc_visit_heaps_lock_held /__w/jax/jax/cpython/Python/gc_free_threading.c:267:14 (python3.13+0x4446ff) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #9 gc_visit_heaps /__w/jax/jax/cpython/Python/gc_free_threading.c:306:11 (python3.13+0x4446ff)
    #10 deduce_unreachable_heap /__w/jax/jax/cpython/Python/gc_free_threading.c:614:5 (python3.13+0x4458db) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #11 gc_collect_internal /__w/jax/jax/cpython/Python/gc_free_threading.c:1126:15 (python3.13+0x4458db)
    #12 gc_collect_main /__w/jax/jax/cpython/Python/gc_free_threading.c:1239:5 (python3.13+0x4458db)
    #13 _Py_RunGC /__w/jax/jax/cpython/Python/gc_free_threading.c:1684:5 (python3.13+0x44730e) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)

 Previous write of size 8 at 0x7fffc2221a38 by thread T68 (mutexes: read M0):
    #0 __tsan_memset  (python3.13+0xda21d) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #1 memset /usr/include/x86_64-linux-gnu/bits/string_fortified.h:59:10 (xla_extension.so+0xa7a60ec) (BuildId: 0cb922163d5a5c8d99583339143c71c4c2e776b1)
    #2 xla::(anonymous namespace)::PyArray_bf_getbuffer(_object*, Py_buffer*, int)::$_0::operator()() const /proc/self/cwd/external/xla/xla/python/py_array.cc:1601:5 (xla_extension.so+0xa7a60ec)
    #3 xla::(anonymous namespace)::PyArray_bf_getbuffer(_object*, Py_buffer*, int) /proc/self/cwd/external/xla/xla/python/py_array.cc:1518:25 (xla_extension.so+0xa7a60ec)
    #4 PyObject_GetBuffer /__w/jax/jax/cpython/Objects/abstract.c:442:15 (python3.13+0x1ba90d) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)
    #5 _PyManagedBuffer_FromObject /__w/jax/jax/cpython/Objects/memoryobject.c:97:9 (python3.13+0x27fdf5) (BuildId: 5001c71441642d7e6efdfc82265e7ee63d55228e)

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par la référence de la C API à l’adresse docs.python.org/3/c-api/typeobj.html, en vous concentrant sur PyBufferProcs.bf_getbuffer, et examinez, pour le contexte, le site d’allocation lié dans Objects/memoryobject.c. Déterminez si l’exigence s’applique uniquement au free-threading mode et documentez la manière dont les implémentations doivent protéger les écritures dans l’argument view ; le travail sera considéré comme terminé lorsque la documentation de l’API indiquera explicitement l’exigence de synchronisation applicable.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
documentation
Type d'issue
Documentation
Difficulté
3/5
Temps estimé
1-2 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.