[mypyc] Design discussion: build-system-neutral C generation CLI (mypyc cgen) for projects not using setuptools

Bắt đầu với entry point python -m mypyc hiện có và các đường dẫn mypycify()/mypyc_build được mô tả trong issue. Giải quyết các câu hỏi còn bỏ ngỏ về vị trí của CLI, các cơ sở của đường dẫn schema, các tùy chọn profile của compiler và mối quan hệ của nó với mypycify; công việc hoàn tất khi thiết kế được thống nhất và command được đề xuất cùng contract JSON có phiên bản được triển khai với coverage cho hành vi đã được tài liệu hóa.

Motivation

mypycify() is currently the only supported entry point for compiling Python to C with mypyc, and it is tightly coupled to setuptools/distutils: it returns setuptools.Extension objects, patches build_ext, and requires setuptools to be importable (on Python 3.12+ distutils itself only exists through setuptools).

Projects that build their C extensions with meson, CMake, Bazel, scikit-build-core, or plain Makefiles cannot use mypyc without either dragging setuptools in as a secondary build backend or reverse-engineering mypyc's implicit conventions (shim generation, runtime file copying, include dirs, compiler flags, shared-lib naming, output paths).

This issue proposes a build-system-neutral CLI that performs mypyc's frontend + C generation and emits a machine-readable description of the extensions to build. The actual C compilation stays with the caller's build system.

Proposed CLI

mypyc cgen [--target-dir DIR] [--output-file FILE] [--multi-file] [--separate] SOURCE...

Behavior:

1. Run mypyc's frontend and C generation (type check + codegen, same as mypyc_build).
2. Write all generated C files into --target-dir (default build/mypyc): per-group .c/.h files, the runtime library (lib-rt), and per-module shims when a shared library is used.
3. Emit a JSON "build info" document to --output-file, or to stdout if not given.

Flags:

Deliberately not CLI flags: optimization/debug levels. The emitted cflags contain only flags required for the generated code to work correctly (warning suppression, feature macros like -DMYPYC_LOG_TRACE, the multi-file /GL- workaround on msvc); they never include optimization or debug levels. The consumer's build system owns those decisions, so we don't duplicate the knob at the CLI boundary. (mypycify keeps passing its own defaults, -O3 -g1, so the setuptools path is unchanged.)

JSON build info (schema v1)

{
  "schema_version": 1,
  "target_dir": "/abs/path/to/build/mypyc",
  "extensions": [
    {
      "module": "pkg.mod__mypyc",
      "out_path": "pkg/mod__mypyc.cpython-314-x86_64-linux-gnu.so",
      "sources": ["pkg/__native_pkg.mod.c", "init.c"],
      "include_dirs": ["/abs/.../mypyc/lib-rt", "/abs/path/to/build/mypyc"],
      "depends": ["pkg/__native_pkg.mod.h"],
      "cflags": ["-Werror", "-Wno-unused-function", "..."],
      "link_args": ["-shared"]
    }
  ]
}

Field semantics:

• schema_version: integer, bumped on incompatible changes.
• target_dir: absolute path all relative paths below are resolved against.
• extensions: one entry per extension module to build.
  • module: full dotted extension module name. For a package __init__ shim this follows the setuptools convention "pkg.__init__".
  • out_path: where the built file must end up, relative to the installation root (the equivalent of site-packages), including the platform extension suffix. Note this is a different base than sources/depends (which are relative to target_dir) — this distinction must be explicit in the schema since downstream tooling will join paths.
  • sources: .c files to compile, relative to target_dir (keeps the JSON portable across machines/containers).
  • include_dirs: absolute paths (they point into the mypyc installation, e.g. lib-rt, which cannot be made relative).
  • depends: headers/source files that must trigger a rebuild when they change, relative to target_dir (already transitively resolved from #include scanning, including cross-group export-table headers).
  • cflags: compiler flags required by the generated code (warnings, feature macros). Never includes optimization or debug levels — those are up to the caller's build system.
  • link_args: platform link flags needed to produce a loadable extension module, derived from Python's configured LDSHARED. May be empty; the caller is always responsible for the platform's standard extension linking (e.g. the Python import library on Windows) and for the Python.h include directory.

How a build system would consume this (meson example)

• run_command() or a custom_target() invokes mypyc cgen --output-file ... once per configuration;
• configure step parses the JSON and feeds each entry into py.extension_module(...) using sources, include_dirs, cflags, link_args;
• depends maps to meson's dependency tracking for incremental rebuilds.

A concrete working example (compiling src/bgm_tv_wiki/ast.py; the shim keeps the importable name, the shared library holds the generated code):

project('bgm-tv-wiki', 'c',
  default_options: ['c_std=c11', 'buildtype=release'],
  meson_version: '>= 1.5.0')

py = import('python').find_installation(pure: false)

py.install_sources(
  'src/bgm_tv_wiki/__init__.py',
  'src/bgm_tv_wiki/py.typed',
  subdir: 'bgm_tv_wiki',
)

# ---- 1. Run cgen at configure time: generate C sources + info.json ----
cgen_dir = meson.current_build_dir() / 'mypyc-cgen'

run_command(py, '-m', 'mypyc', 'cgen',
  '--target-dir', cgen_dir,
  '--output-file', cgen_dir / 'info.json',
  meson.project_source_root() / 'src' / 'bgm_tv_wiki' / 'ast.py',
  check: true)

# ---- 2. Extract build parameters from info.json (meson can't parse JSON) ----
json_get = '''
import json, sys
info = json.load(open(sys.argv[1]))
ext = next(e for e in info["extensions"] if e["module"] == sys.argv[2])
print("\\n".join(ext[sys.argv[3]]))
'''

cgen_info = cgen_dir / 'info.json'
native = 'bgm_tv_wiki.ast__mypyc'

native_sources = run_command(py, '-c', json_get, cgen_info, native, 'sources',
  check: true).stdout().strip().split('\n')
native_include_dirs = run_command(py, '-c', json_get, cgen_info, native, 'include_dirs',
  check: true).stdout().strip().split('\n')
native_cflags = run_command(py, '-c', json_get, cgen_info, native, 'cflags',
  check: true).stdout().strip().split('\n')
native_link = run_command(py, '-c', json_get, cgen_info, native, 'link_args',
  check: true).stdout().strip().split('\n')

# sources are relative to target_dir; make them absolute
native_sources = [cgen_dir / s for s in native_sources]
# include_dirs are absolute paths (into the mypyc installation); turn them into -I flags
foreach d : native_include_dirs
  native_cflags += ['-I' + d]
endforeach

# ---- 3. Build the extensions ----
# native: compile the mypyc-generated C (include_dirs already covers lib-rt
# and the cgen output directory). No py.dependency() needed — meson's python
# module provides Python.h and the import library automatically.
py.extension_module(
  'ast__mypyc',
  native_sources,
  subdir: 'bgm_tv_wiki',
  install: true,
  c_args: native_cflags,
  link_args: native_link,
)

# shim: forwards to the native module; only needs Python.h
shim_sources = run_command(py, '-c', json_get, cgen_info, 'bgm_tv_wiki.ast', 'sources',
  check: true).stdout().strip().split('\n')
py.extension_module(
  'ast',
  [cgen_dir / s for s in shim_sources],
  subdir: 'bgm_tv_wiki',
  install: true,
)

Why a CLI (not a Python API)

• Process isolation: the caller's build system can invoke it as a tool without importing mypyc into its own process, and without setuptools installed at all (the CLI path never touches distutils/setuptools).
• The JSON document is a stable, versioned contract between mypyc and any build backend; a Python API would tie callers to mypyc internals and Python packaging conventions.

Open questions

1. Naming/placement: mypyc cgen as a subcommand of the existing python -m mypyc entry point, or a separate console script?
2. Is the schema stable enough to commit to, particularly the "two path bases" convention (sources/depends vs target_dir vs out_path)?
3. Should there be an escape hatch for compiler-profile knobs (e.g. --cflags passthrough), or is "post-edit the JSON" sufficient?
4. Long-term relationship with mypycify: keep both (setuptools users keep mypycify; everyone else uses cgen), or eventually re-implement mypycify on top of cgen?

Happy to work on the implementation once the design is agreed upon.