Modernization roadmap: multi-phase init, stable ABI, free-threading, GC support, deprecated-API cleanup (discussion)

Nobody has claimed this yet.

## Summary

A discussion / tracking issue for five modernization topics that are all feasible on zstandard's current codebase but involve design trade-offs better agreed up-front before PR work begins. Each sub-topic could be its own issue + PR if you'd prefer smaller surfaces; grouping here so the full landscape is visible.

The five topics:

1. Multi-phase module init (sub-interpreter readiness).
2. Stable ABI (`abi3`) adoption — blocked only by one private API.
3. `Py_MOD_GIL_NOT_USED` is declared but structurally premature.
4. GC support missing on 15 types that store `PyObject *` fields.
5. Deprecated-API cleanup — small, mechanical.

## 1. Multi-phase module init

**Current state:** `m_size = -1` + single-phase init blocks sub-interpreter use. 19 global `PyTypeObject *`, 1 global `PyObject *ZstdError`, and a `features` set all live in C globals rather than per-module state.

**Migration path:** `PyModuleDef_Slot` array with `Py_mod_create` / `Py_mod_exec`; types and other globals move into a per-module state struct, looked up via `PyType_GetModuleByDef(...)`.

**Effort:** Medium. The global-to-state migration is the bulk of the work; slot adoption is mechanical once state is in place.

## 2. Stable ABI (`abi3`) feasibility

**Current state:** Close to feasible — only one real blocker.

- Heap types already in place via `PyType_FromSpec`.
- `pythoncapi_compat.h` already included.
- ~6 `PyBytes_AS_STRING` + list/tuple macros → replace with their non-limited equivalents (mechanical).
- `_PyBytes_Resize` is the actual blocker — private API used for copy-avoidance in `safe_pybytes_resize`.

**Target:** `Py_LIMITED_API = 0x030A0000` (3.10 floor); guard `_PyBytes_Resize` with `#ifndef Py_LIMITED_API`, falling back to a copy; replace unlimited macros with limited-API equivalents.

**Impact:** Wheel matrix collapses from per-minor-version to one `abi3` wheel per platform. Minor perf cost on `_PyBytes_Resize` fallback path (one extra copy per resize), which is only hit on the abi3 build.

## 3. `Py_MOD_GIL_NOT_USED` — premature?

The declaration asserts free-threading safety. The current structural reality:

- 19 global `PyTypeObject *` (not per-interpreter).
- No per-object locking (no `Py_BEGIN_CRITICAL_SECTION`).
- Single-phase init (see topic 1).
- CFFI backend has no FT story.

**Options:**
- **(A)** Gate the declaration behind topic 1 + a critical-section audit.
- **(B)** `#if 0` it so maintainers flip it intentionally after the structural work lands.

Today the declaration is a promise the code doesn't fully keep. Free-threaded CPython users who load zstandard into a shared-type program can hit races on the global type pointers during multi-interpreter or multi-thread setup/teardown.

## 4. GC support missing — 15 types

15 of the 19 heap types store `PyObject *` fields (compressor refs, source refs, result refs, etc.) without `Py_TPFLAGS_HAVE_GC`. Reference cycles through these fields are uncollectible.

**Not actively leaking today** — the fields are typically short-lived — but user code that creates a cycle through one of these fields (e.g., a callback closure that retains the compressor) leaks until process exit.

**Fix:** Add `Py_TPFLAGS_HAVE_GC` to each spec; implement `tp_traverse` visiting every `PyObject *` field; `tp_clear` using `Py_CLEAR` on each.

**Interaction:** The separately-filed heap-type refcount issue migrates 4 factories from `PyObject_New` to `tp_alloc`; `tp_alloc` already handles GC registration if the type is flagged `HAVE_GC`. The two changes compose cleanly.

## 5. Deprecated-API cleanup

Small, mechanical; can be done in a single PR independent of the above.

- Remove 9 dead `#if PY_VERSION_HEX < 0x03090000` guards in `c-ext/bufferutil.c` — the supported-Python floor is 3.9.
- Drop `#include "structmember.h"` (deprecated in 3.12; `pythoncapi_compat.h` supplies it).
- `PyModule_AddObject` → `PyModule_AddObjectRef` — covered in the separately-filed OOM-hardening issue.
- 16 `PyObject_CallObject` → `PyObject_CallNoArgs` / `PyObject_Call` — clarity; `CallObject` is still stable ABI, so this is cosmetic.
- `PY_SSIZE_T_CLEAN` is a no-op from 3.13; can be removed alongside the 3.13+ bump if/when it happens.
- `pythoncapi_compat.h` is **included but largely unused** — `PyObject_HasAttrStringWithError`, `Py_XSETREF`, and others in it could replace existing less-robust patterns (the `HasAttrString` → `HasAttrStringWithError` migration is in the separately-filed `MemoryError`-swallowing issue).

## Filing as one discussion vs. 5 issues

Grouped here because these decisions are correlated: "where does zstandard want to be in 2-3 Python releases?". Multi-phase init is a precondition for (3) meaningfully; abi3 is orthogonal but valuable to plan alongside; GC and deprecated-API are standalone.

Happy to split into 5 separate issues + staged PRs if you prefer smaller surfaces for review. Also happy to take on any of these as a PR once you've indicated priority — my suggestion would be topic 5 first (small mechanical cleanup, low risk), topic 2 next (abi3 is high-leverage at modest cost), topic 4 (GC support), and save topics 1 + 3 for last since they interact and benefit from an aligned design.

## Methodology

Found via [cext-review-toolkit](https://github.com/devdanzin/cext-review-toolkit) — the module-state, stable-abi, free-threading, type-slot, and version-compat scanners collectively covered these topics. All five items are static observations about the codebase's current shape vs. modern CPython C-extension conventions; no live reproducer is applicable.

*Discovery, root-cause analysis, and issue drafting were performed by [Claude Code](https://www.anthropic.com/claude-code) and reviewed by a human before filing.*

## Full report

Complete multi-agent analysis (48 FIX findings across 13 categories, plus a reproducer appendix): https://gist.github.com/devdanzin/b86039ac097141579590c1a0f3a43605

Start by separating the five topics into independently scoped work, using c-ext/bufferutil.c for the deprecated-API cleanup and the module/type definitions for the broader initialization, ABI, GIL, and GC discussions. Review the linked methodology and existing issue dependencies first. Done means maintainers have agreed on priorities, design boundaries, and separate actionable issues or PR scopes.

Written by the indexing model from the issue text.