Pass opaque state to PyContext_AddWatcher callback

Feature or enhancement

Proposal:

I would like to change the Context watcher C API (new to v3.14, see #119333) to pass an opaque pointer to the callback. Specifically, change:

https://github.com/python/cpython/blob/e8bb05394164e7735f7a9de80a046953606a38eb/Include/cpython/context.h#L39-L55

to:

/*
 * Context object watcher callback function.  arg is the same pointer passed to
 * PyContext_AddWatcher when the callback was registered.  The object passed to
 * the callback is event-specific; see PyContextEvent for details.
 *
 * if the callback returns with an exception set, it must return -1. Otherwise
 * it should return 0
 */
typedef int PyContext_WatchCallback(
    void *arg, PyContextEvent event, PyObject *obj);

/*
 * Register a per-interpreter callback that will be invoked for context events.
 * arg is an optional opaque pointer that is passed back to the callback; it
 * can be used to manage state if desired.
 *
 * Returns a handle that may be passed to PyContext_ClearWatcher on success,
 * or -1 and sets and error if no more handles are available.
 */
PyAPI_FUNC(int) PyContext_AddWatcher(
    PyContext_WatchCallback *callback, void *arg);

The original idea was for the callback to get any required state from the current context, but:

• The config/state is not guaranteed to be available in the context. For example, the Python code can do ctx = contextvars.Context() instead of ctx = contextvars.copy_context(). Or it can do copy_context(), but early during initialization and squirrel the context away for later use. The latter seems plausible for 3rd party libraries (where the user might not have much control).
• The same watcher callback cannot be registered multiple times simultaneously, each with its own config/state. This came up while I was refactoring TestContextObjectWatchers to add some more tests for a change I’m working on.

I’m not sure how likely either is to come up in normal use, but I think it is worth changing the design before the 3.14 release cements it.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/v3-14a1-design-limitations-of-pycontext-addwatcher/68177/4

cc @fried

Linked PRs

• gh-127140
• gh-127247

Start with Include/cpython/context.h at the Context watcher declarations and review Lib/test/test_capi/test_watchers.py, especially TestContextObjectWatchers. Check the linked PRs before starting, then verify that the registered opaque pointer reaches the callback and that the watcher tests cover the updated API and pass.