PyPI sdist missing C++ source — cannot build native libraries from source
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## 🐞Describing the bug
The PyPI source distribution (sdist) does not include the C++ source directories needed to build the native libraries (`libmilstoragepython`, `libmodelpackage`, `kmeans1d/_core`). Building from the sdist produces a wheel without these libraries, so model conversion crashes at runtime.
The Git repository contains everything needed — `CMakeLists.txt`, `deps/`, `mlmodel/`, `cmake/`, `modelpackage/`, `milstoragepython/` — but these are excluded from the sdist tarball. The `setup.py` packages the resulting `.so` files via `package_data`, but since the CMake build never runs during `pip install`, they don't exist.
## Stack Trace
WARNING: No module named 'coremltools.libmilstoragepython'
WARNING: No module named 'coremltools.libcoremlpython'
RuntimeError: BlobWriter not loaded
## To Reproduce
pip install coremltools --no-binary coremltools
python -c "
import coremltools as ct
import torch
model = torch.nn.Linear(10, 5)
traced = torch.jit.trace(model, torch.randn(1, 10))
ct.convert(traced) # RuntimeError: BlobWriter not loaded
"
## System environment (please complete the following information):
- coremltools version: 8.3 and 9.0 (likely all versions)
- OS (e.g. MacOS version or Linux type): Linux x86_64 (RHEL 9), but affects any platform when building from sdist
- Any other relevant version information (e.g. PyTorch or TensorFlow version): Python 3.12, PyTorch 2.6.0
## Additional context
Building from the Git repository with CMake works perfectly — all three native libraries compile and model conversion succeeds. The published wheels on PyPI (both macOS and manylinux) include the native libraries and work fine — the issue is only with the sdist.
The fix would be to include the C++ source directories in `MANIFEST.in` (or equivalent) and wire the CMake build into the PEP 517 build backend so `pip install` from the sdist produces a working package.
Contributor guide
Research direction
Start with setup.py, MANIFEST.in, CMakeLists.txt, and the PEP 517 build configuration, then reproduce the failure with pip install --no-binary coremltools from the sdist. Verify that the sdist contains the listed C++ source directories, that installation builds the native libraries, and that the provided model-conversion command no longer raises BlobWriter not loaded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100