airbytehq / airbytehq/PyAirbyte

build: Add Cython compilation support for wheel obfuscation

Abierto
#979 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
344
Forks
77
Merge medio
1 d 11 h
PR fusionados (30 d)
35

Descripción

## Summary

Add Cython compilation to the PyAirbyte build pipeline so that PyPI-published wheels contain compiled `.so`/`.pyd` extensions instead of readable `.py` source files, providing basic code obfuscation for distribution from the private repo.

## Context

This pattern has been prototyped in the airbyte-ops-mcp and connector-builder-mcp repos:
- **Ops MCP PR**: https://github.com/airbytehq/airbyte-ops-mcp/pull/343
- **Connector Builder MCP PR**: https://github.com/airbytehq/connector-builder-mcp/pull/208
- **Related ops-mcp issue**: https://github.com/airbytehq/airbyte-ops-mcp/issues/346

The approach uses:
- `hatch-cythonize` build hook (maintained fork of hatch-cython) to compile `.py` → `.c` → `.so` during wheel builds
- Build-time injection of Cython config (not stored in `pyproject.toml`) to avoid breaking `uv sync` / editable installs
- Post-build source stripping to remove `.py` files from wheels where compiled `.so` counterparts exist
- Exclusion of modules that break under Cython compilation (e.g., MCP/Pydantic presentation layers)

## Key Learnings from Ops MCP Implementation

1. **Cython config must NOT be in `pyproject.toml`** — it runs during editable installs (`uv sync`), breaking CI workflows. Solution: inject at build time via a script.
2. **Pydantic/FastMCP modules must be excluded** — Cython-compiled functions (`cyfunction`) don't expose type annotations the same way, breaking Pydantic schema generation.
3. **`__init__.py` files must be excluded** — required for Python package structure.
4. **Platform-specific wheels** — must build per OS (Linux, macOS) and Python version (3.10, 3.11, 3.12).

## Action Items

- [ ] Evaluate whether this pattern should be applied to PyAirbyte
- [ ] If yes, adapt the `enable_cython_build.py` and `strip_source_from_wheel.py` scripts for PyAirbyte's package structure
- [ ] Identify which modules (if any) need to be excluded from compilation
- [ ] Integrate with existing publish/release workflows

Devin session: https://app.devin.ai/sessions/7c8622f9452141d1bbad93f74079278f
Requested by: @aaronsteers

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.