`#[pyclass(storage = ...)]` for fine-grained control of pyclass behaviour

Context

We currently have a few ideas in the works which all depend on improved (or modified) #[pyclass] semantics:

• Make #[pyclass(frozen)] the default as per #4369
• Introduction of the PyClassGuard(Mut) types in #5233 to the user-facing API
• Proposed deprecation of #[pyclass(unsendable)] as per #5407
• Advanced borrow error tracking in #5412
• (Maybe) borrow from one #[pyclass] in another as per #1089
• (Maybe) take ownership of smart pointers to data - Arc<T>, Box<T>? (Related to #4887)

PyClassObjectContents

All of these features are in one way or another heavily coupled to our PyClassObjectContents type:

#[repr(C)]
pub(crate) struct PyClassObjectContents<T: PyClassImpl> {
    pub(crate) value: ManuallyDrop<UnsafeCell<T>>,
    pub(crate) borrow_checker: <T::PyClassMutability as PyClassMutability>::Storage,
    pub(crate) thread_checker: T::ThreadChecker,
    // there are also `dict` and `weakref` fields which we should really be phasing out as per #4253
}

These three fields are very coupled to the existing functionality (and its limitations):

• value requires that we hold the state directly inside the Python object
• borrow_checker is either populated or zero side depending on if #[pyclass(frozen)]
• thread_checker is used to implement #[pyclass(unsendable)]

I think to evolve the PyO3 API it feels like we need to replace these with some more pluggable functionality.

PyClassStorage

I think we might want to consider a PyClassStorage trait which handles the lifecycle of the object. It's probably something like this:

trait PyClassStorage<T>: Send + Sync {
    type Guard<'a>: Deref<Target = T> where Self: 'a;
    type MutableGuard<'a>: DerefMut<Target = T> where Self: 'a;
    fn borrow(&self, py: Python<'_>) -> Result<Self::Guard<'_>, PyBorrowError>;
    fn borrow_mut(&self, py: Python<'_>) -> Result<Self::MutableGuard<'_>, PyBorrowMutError>;

    fn initialize(&mut self, value: /* TBC */);  // not necessarily T here, e.g. what if it's `Arc<T>`
    fn clear(&mut self);
}

It might be that to encode frozen semantics, we would need to split half of this trait off into a PyClassMutableStorage<T> trait which is only available for mutable methods.

I think this trait (or traits) could be implented for a bunch of types like:

• T, Box<T> and Arc<T> (for frozen classes)
• PyCell<T> (which would do the borrow tracking - interestingly Box<T> would also be a reasonable value to put inside a PyCell)
• Unsendable<Storage> (wraps some other storage to forbid accessing the object from other threads)
• Mutex<T> (maybe optionally wrapped in Arc) for types which lock instead of do our current borrow tracking

Users could select which one of these they want to use with a #[pyclass(storage = ...)] option. (I originally suggested this option in https://github.com/PyO3/pyo3/pull/5233#issuecomment-3089194417.)

A nice think about exposing this as a trait is it makes it possible for users to implement their own storage types, if they have some very bespoke situation.

I think there's a lot to play around with here and I think it'd be a powerful improvement for PyO3.

Questions

A ton of stuff I don't yet have answers to:

• Can we start merging this option as "experimental" so we can play with semantics and review bits of this incrementally instead of in one mega PR?
• If we can solve #1089, presumably that's not ever going to map cleanly to the user-selected storage type. Does this imply we need to use an enum (or dynamic dispatch) inside the PyClassObjectContents to allow for multiple possible storages in the same type?
• Some "storage" cases like T, Box<T> and Arc<T> for frozen classes seem like they should be interchangeable. Does this again imply we want to allow multiple possible storages / dynamic dispatch?
• Should we avoid supporting Box<T> and Arc<T> and leave users to explicitly use those within their pyclass (e.g. #[pyclass] struct MyClass(Arc<MyClassInner>))?

Start by reading the PyClassObjectContents definition and the linked discussions in #4369, #5233, #5407, #5412, #1089, and #4887. Done would require an agreed design for experimental #[pyclass(storage = ...)] semantics, including storage traits, supported implementations, and interaction with borrowing, threading, and object lifecycle.