python / python/cpython

Data race reading `UnicodeDecodeError.start`/`end` in `__str__` under free-threading

Open
#154,930 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

interpreter-core type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

On a free-threaded build, UnicodeDecodeError.__str__ reads the start and end members with plain (non-atomic) loads, while assigning to exc.start / exc.end goes through PyMember_SetOne, which does an atomic store. So calling str(exc) on a shared UnicodeDecodeError concurrently with setting its start/end is a data race on those Py_ssize_t members.

start and end are exposed as Py_T_PYSSIZET members:
https://github.com/python/cpython/blob/22a6c51c94a4fde986b8964f1d36d5ec3ac20dcc/Objects/exceptions.c#L3758-L3761

Plain read in UnicodeDecodeError_str:
https://github.com/python/cpython/blob/22a6c51c94a4fde986b8964f1d36d5ec3ac20dcc/Objects/exceptions.c#L3947-L3966

Reproducer:

from threading import Thread

shared_exc = UnicodeDecodeError('utf-8', b'\xff\xfe\xfa', 1, 2, 'invalid start byte')

def chain1_thread():
    for i in range(20000):
        try:
            shared_exc.start = i % 3
        except Exception:
            pass

def chain2_thread():
    for _ in range(20000):
        try:
            str(shared_exc)
        except Exception:
            pass

N_C1 = 4
N_C2 = 8
threads  = [Thread(target=chain1_thread) for _ in range(N_C1)]
threads += [Thread(target=chain2_thread) for _ in range(N_C2)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Report:

WARNING: ThreadSanitizer: data race (pid=656224)
  Read of size 8 at 0x7fffb63734e8 by thread T5:
    #0 UnicodeDecodeError_str /cpython/Objects/exceptions.c:3948:29 
    #1 PyObject_Str /cpython/Objects/object.c:826:11 
    #2 unicode_vectorcall /cpython/Objects/unicodeobject.c:14279:16 
    #3 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11 
    #4 PyObject_Vectorcall /cpython/Objects/call.c:327:12 
    #5 _Py_VectorCallInstrumentation_StackRefSteal /cpython/Python/ceval.c:768:11 
    #6 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:1906:35

  Previous atomic write of size 8 at 0x7fffb63734e8 by thread T3:
    #0 PyMember_SetOne /cpython/Python/structmember.c 
    #1 member_set /cpython/Objects/descrobject.c:239:12 
    #2 _PyObject_GenericSetAttrWithDict /cpython/Objects/object.c:2049:19 
    #3 PyObject_GenericSetAttr /cpython/Objects/object.c:2120:12 
    #4 PyObject_SetAttr /cpython/Objects/object.c:1533:15 
    #5 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:12057:27

SUMMARY: ThreadSanitizer: data race/cpython/Objects/exceptions.c:3948:29 in UnicodeDecodeError_str
CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-155255

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Objects/exceptions.c at the Py_T_PYSSIZET member declarations and UnicodeDecodeError_str, then run the supplied threading reproducer on a free-threaded build and inspect the TSAN report. Done means the reported race is addressed and relevant CPython tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.