python / python/cpython

Race condition in `itertools.islice` under free-threading

未關閉
#151,409 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

extension-modules topic-free-threading type-bug
主要語言
Python
星號
77.2k
分支
36k
PR 合併指標
PR 指標待擷取

描述

Bug report

Bug description:

islice_next reads and writes three fields -- lz->cnt, lz->next, and lz->it -- and do operations on them without a critical section.

https://github.com/python/cpython/blob/d986124d83465190987357f987ee24bd7a817cac/Modules/itertoolsmodule.c#L1628-L1661

Two threads calling next on the same islice object concurrently can race:

  • Both read lz->cnt = N < lz->next, both execute the skip-loop body for the same slot, advancing the underlying iterator twice for one step. In that case, the step arithmetic gets corrupted.
  • Both read lz->cnt = N < stop, both pass the stop check, both call iternext(it) and return an item -- the total number of yielded items exceeds stop.
  • There's a race on lz->cnt++ and lz->next += step: both threads can read the same value, then both can increment locally, and both store the same result. After that, the counter is permanently incorrect (such that subsequent skip/stop decisions use a wrong baseline).
  • One of the threads reaches exhaustion (goto empty) and calls Py_CLEAR(lz->it), which sets lz->it = NULL and DECREF's the iterator, which frees it (since each thread has ready it = lz->it. and not INCREF'd it). Another thread can have read lz->it before the clear and be in the middle of iternext(it) on the now-freed object in which case there is a use-after-free.

chain_next for example wraps its body in Py_BEGIN_CRITICAL_SECTION(op), which I believe is what islice should do as well.

https://github.com/python/cpython/blob/d986124d83465190987357f987ee24bd7a817cac/Modules/itertoolsmodule.c#L1937-L1945

Reproducer

import itertools
import threading

STOP = 100
NTHREADS = 8

data = iter(range(STOP + NTHREADS * 2))
sl = itertools.islice(data, STOP)

results: list[int] = []
lock = threading.Lock()

def consume() -> None:
    while True:
        v = next(sl, None)
        if v is None:
            break
        with lock:
            results.append(v)

threads = [threading.Thread(target=consume) for _ in range(NTHREADS)]
for t in threads: t.start()
for t in threads: t.join()

The reproducer, on a free-threaded build, gets reports like this one.

WARNING: ThreadSanitizer: data race (pid=49886)
  Read of size 8 at 0x00030275f8b0 by thread T2:
    #0 islice_next itertoolsmodule.c:1663 (python.exe:arm64+0x10042bce4)
    #1 builtin_next bltinmodule.c:1770 (python.exe:arm64+0x10028a900)
    #2 cfunction_vectorcall_FASTCALL methodobject.c:449 (python.exe:arm64+0x1001379dc)
    #3 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100090e80)
    #4 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090e80)
...
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-151410

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

從 Modules/itertoolsmodule.c 中的 islice_next 開始,並比較 chain_next 對臨界區段的處理。使用 free-threaded CPython 建置執行隨附的多執行緒重現程式,並檢查相關的 itertools 測試。當並行呼叫 next() 不再觸發回報的競爭條件,也不再超出 islice 的 stop 邊界時,即表示完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
backend
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
描述清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。