airbytehq / airbytehq/PyAirbyte

build: Add Cython compilation support for wheel obfuscation

Đang mở
#979 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

## Summary

Add Cython compilation to the PyAirbyte build pipeline so that PyPI-published wheels contain compiled `.so`/`.pyd` extensions instead of readable `.py` source files, providing basic code obfuscation for distribution from the private repo.

## Context

This pattern has been prototyped in the airbyte-ops-mcp and connector-builder-mcp repos:
- **Ops MCP PR**: https://github.com/airbytehq/airbyte-ops-mcp/pull/343
- **Connector Builder MCP PR**: https://github.com/airbytehq/connector-builder-mcp/pull/208
- **Related ops-mcp issue**: https://github.com/airbytehq/airbyte-ops-mcp/issues/346

The approach uses:
- `hatch-cythonize` build hook (maintained fork of hatch-cython) to compile `.py` → `.c` → `.so` during wheel builds
- Build-time injection of Cython config (not stored in `pyproject.toml`) to avoid breaking `uv sync` / editable installs
- Post-build source stripping to remove `.py` files from wheels where compiled `.so` counterparts exist
- Exclusion of modules that break under Cython compilation (e.g., MCP/Pydantic presentation layers)

## Key Learnings from Ops MCP Implementation

1. **Cython config must NOT be in `pyproject.toml`** — it runs during editable installs (`uv sync`), breaking CI workflows. Solution: inject at build time via a script.
2. **Pydantic/FastMCP modules must be excluded** — Cython-compiled functions (`cyfunction`) don't expose type annotations the same way, breaking Pydantic schema generation.
3. **`__init__.py` files must be excluded** — required for Python package structure.
4. **Platform-specific wheels** — must build per OS (Linux, macOS) and Python version (3.10, 3.11, 3.12).

## Action Items

- [ ] Evaluate whether this pattern should be applied to PyAirbyte
- [ ] If yes, adapt the `enable_cython_build.py` and `strip_source_from_wheel.py` scripts for PyAirbyte's package structure
- [ ] Identify which modules (if any) need to be excluded from compilation
- [ ] Integrate with existing publish/release workflows

Devin session: https://app.devin.ai/sessions/7c8622f9452141d1bbad93f74079278f
Requested by: @aaronsteers

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.