python / python/cpython

`groupby_next` data race on free-threaded builds

Open
#150,791 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules topic-free-threading type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

Two threads calling next on the same groupby object concurrently in a free-threaded build race on the currgrouper field, corrupting the iterator's internal state and producing AttributeError on slot accesses of live objects.

groupby_next has no Py_BEGIN_CRITICAL_SECTION guard. The first thing it does is write gbo->currgrouper = NULL:

https://github.com/python/cpython/blob/c5516e7e371f7b273eb37c7b65f14ef14ee81f11/Modules/itertoolsmodule.c#L531-L540

Later, after the loop exits, it calls _grouper_create, which writes parent->currgrouper = igo:

https://github.com/python/cpython/blob/c5516e7e371f7b273eb37c7b65f14ef14ee81f11/Modules/itertoolsmodule.c#L624-L637

If Thread A is past line 633 (just stored the new grouper pointer) while Thread B is at line 537 (about to store NULL), Thread B overwrites the pointer Thread A just wrote. Both are plain pointer stores with no synchronisation.


The following script reproduces the condition under which this happens.

import itertools, threading

class K:
    __slots__ = ("v",)
    def __init__(self, v): self.v = v
    def __eq__(self, o): return isinstance(o, K) and self.v == o.v
    def __hash__(self): return hash(self.v)

def consume(g):
    try:
        while True:
            _, _ = next(g)
    except StopIteration:
        pass

keys = [K(i) for i in range(500_000)]
g = itertools.groupby(keys)
threads = [threading.Thread(target=consume, args=(g,)) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

On a free-threaded build, this results in the following Python logs...

Exception in Thread-15 (consume):
  File "<python-input-2>", line 6, in __eq__
    def __eq__(self, o): return isinstance(o, K) and self.v == o.v
                                                     ^^^^^^
AttributeError: 'K' object has no attribute 'v'

... and the following TSan logs.

WARNING: ThreadSanitizer: data race (pid=20464)
  Write of size 8 at 0x0003026e4a88 by thread T2:
    #0 groupby_next itertoolsmodule.c:537 (python.exe:arm64+0x10042a8b8)
    #1 builtin_next bltinmodule.c:1770

  Previous write of size 8 at 0x0003026e4a88 by thread T1:
    #0 _grouper_create itertoolsmodule.c:633 (python.exe:arm64+0x10042ac5c)
    #1 groupby_next itertoolsmodule.c:570 (python.exe:arm64+0x10042ac5c)
    #2 builtin_next bltinmodule.c:1770
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-150792

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Modules/itertoolsmodule.c at groupby_next and _grouper_create, then run the provided concurrent reproducer on a free-threaded build with ThreadSanitizer. Done means the reproducer no longer reports the currgrouper race or produces the described AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.