3.11: _PyUnicode_Equal call sites lack error handling

Nobody has claimed this yet.

(Found while looking at https://github.com/python/cpython/issues/98783)

In https://github.com/python/cpython/commit/81c72044a181dbbfbf689d7a977d0d99090f26a8, several uses of _PyUnicode_EqualToASCIIId, which cannot fail, were replaced with _PyUnicode_Equal, which can fail: it calls PyUnicode_READY() in Python 3.11, and returns -1 on failure.

In Python 3.12, this is not a problem: _PyUnicode_Equal cannot fail, since it does not need to call PyUnicode_READY() since the wstr APIs were removed in https://github.com/python/cpython/commit/f9c9354a7a173eaca2aa19e667b5cf12167b7fed. But it is theoretically a problem in 3.11.

In 3.11, git grep "_PyUnicode_Equal(" turns up the following:

Include/cpython/unicodeobject.h:PyAPI_FUNC(int) _PyUnicode_Equal(PyObject *, PyObject *);
Modules/_pickle.c:            use_newobj_ex = _PyUnicode_Equal(name, &_Py_ID(__newobj_ex__));
Modules/_pickle.c:                use_newobj = _PyUnicode_Equal(name, &_Py_ID(__newobj__));
Objects/longobject.c:    else if (_PyUnicode_Equal(byteorder, &_Py_ID(little)))
Objects/longobject.c:    else if (_PyUnicode_Equal(byteorder, &_Py_ID(big)))
Objects/longobject.c:    else if (_PyUnicode_Equal(byteorder, &_Py_ID(little)))
Objects/longobject.c:    else if (_PyUnicode_Equal(byteorder, &_Py_ID(big)))
Objects/typeobject.c:    if (mod != NULL && !_PyUnicode_Equal(mod, &_Py_ID(builtins)))
Objects/typeobject.c:        if (_PyUnicode_Equal(name, &_Py_ID(__dict__))) {
Objects/typeobject.c:        if (_PyUnicode_Equal(name, &_Py_ID(__weakref__))) {
Objects/typeobject.c:        if ((ctx->add_dict && _PyUnicode_Equal(slot, &_Py_ID(__dict__))) ||
Objects/typeobject.c:            (ctx->add_weak && _PyUnicode_Equal(slot, &_Py_ID(__weakref__))))
Objects/typeobject.c:            if (!_PyUnicode_Equal(slot, &_Py_ID(__qualname__)) &&
Objects/typeobject.c:                !_PyUnicode_Equal(slot, &_Py_ID(__classcell__)))
Objects/typeobject.c:    if (mod != NULL && !_PyUnicode_Equal(mod, &_Py_ID(builtins)))
Objects/typeobject.c:        _PyUnicode_Equal(name, &_Py_ID(__class__)))
Objects/typeobject.c:        if (_PyUnicode_Equal(name, &_Py_ID(__class__))) {
Objects/unicodeobject.c:_PyUnicode_Equal(PyObject *str1, PyObject *str2)
Python/ceval.c:            int res = _PyUnicode_Equal(left, right);
Python/errors.c:        if (!_PyUnicode_Equal(modulename, &_Py_ID(builtins)) &&
Python/errors.c:            !_PyUnicode_Equal(modulename, &_Py_ID(__main__))) {
Python/pythonrun.c:        if (!_PyUnicode_Equal(modulename, &_Py_ID(builtins)) &&
Python/pythonrun.c:            !_PyUnicode_Equal(modulename, &_Py_ID(__main__)))

Broken down:

_pickle.c
- Could the result of _PyObject_LookupAttr(callable, &_Py_ID(__name__), &name) be unready?
longobject.c
- Safe because argument clinic calls PyUnicode_READY()
typeobject.c
- ctx->slots could be an arbitrary tuple of potentially-unready strings, so there should be some call to PyUnicode_READY() at or before type_new_visit_slots.
ceval.c
- Already has the error checking (though it could be removed in 3.12!)
errors.c and pythonrun.c:
- Arbitrary result of PyObject_GetAttr(exc_type, &_Py_ID(__module__)); might not be _READY()?

The good news is that this would be hard to run into in practice: it requires both using the old deprecated wstr APIs and running into a memory error during PyUnicode_READY().

cc @ericsnowcurrently @methane

Start with git grep for _PyUnicode_Equal on the Python 3.11 branch, then read the listed call sites in Modules/_pickle.c, Objects/longobject.c, Objects/typeobject.c, Python/ceval.c, Python/errors.c, and Python/pythonrun.c. Trace whether each input can be an unready Unicode object and how PyUnicode_READY() failures are propagated. Done means the affected call sites have safe error handling or a verified readiness guarantee.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment