airbytehq / airbytehq/PyAirbyte
build: Add Cython compilation support for wheel obfuscation
- Vorherrschende Sprache
- Python
- Sterne
- 344
- Forks
- 77
- Ø Merge
- 1 T. 11 Std.
- Gemergte PRs (30 T.)
- 35
Beschreibung
## Summary
Add Cython compilation to the PyAirbyte build pipeline so that PyPI-published wheels contain compiled `.so`/`.pyd` extensions instead of readable `.py` source files, providing basic code obfuscation for distribution from the private repo.
## Context
This pattern has been prototyped in the airbyte-ops-mcp and connector-builder-mcp repos:
- **Ops MCP PR**: https://github.com/airbytehq/airbyte-ops-mcp/pull/343
- **Connector Builder MCP PR**: https://github.com/airbytehq/connector-builder-mcp/pull/208
- **Related ops-mcp issue**: https://github.com/airbytehq/airbyte-ops-mcp/issues/346
The approach uses:
- `hatch-cythonize` build hook (maintained fork of hatch-cython) to compile `.py` → `.c` → `.so` during wheel builds
- Build-time injection of Cython config (not stored in `pyproject.toml`) to avoid breaking `uv sync` / editable installs
- Post-build source stripping to remove `.py` files from wheels where compiled `.so` counterparts exist
- Exclusion of modules that break under Cython compilation (e.g., MCP/Pydantic presentation layers)
## Key Learnings from Ops MCP Implementation
1. **Cython config must NOT be in `pyproject.toml`** — it runs during editable installs (`uv sync`), breaking CI workflows. Solution: inject at build time via a script.
2. **Pydantic/FastMCP modules must be excluded** — Cython-compiled functions (`cyfunction`) don't expose type annotations the same way, breaking Pydantic schema generation.
3. **`__init__.py` files must be excluded** — required for Python package structure.
4. **Platform-specific wheels** — must build per OS (Linux, macOS) and Python version (3.10, 3.11, 3.12).
## Action Items
- [ ] Evaluate whether this pattern should be applied to PyAirbyte
- [ ] If yes, adapt the `enable_cython_build.py` and `strip_source_from_wheel.py` scripts for PyAirbyte's package structure
- [ ] Identify which modules (if any) need to be excluded from compilation
- [ ] Integrate with existing publish/release workflows
Devin session: https://app.devin.ai/sessions/7c8622f9452141d1bbad93f74079278f
Requested by: @aaronsteers
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.