lablup / lablup/mlxcel

feat: distribute the mlxcel binary via pip so `pip install` yields a runnable managed mode

Open
#416 0 comments 1 reaction 1 assignee Claimed by @inureyes View on GitHub
priority:medium status:ready type:enhancement
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Summary

Follow-up to #407 (the Phase 1 pure-Python `mlxcel` client, merged). Today `pip install mlxcel` gives you the client but NOT a runnable managed mode: `ManagedServer._find_binary` discovers an externally installed `mlxcel` executable (order: `binary=` argument, then `MLXCEL_BIN`, then `mlxcel` on `PATH`), which the user must install separately via Homebrew (`brew install lablup/tap/mlxcel`, macOS arm64 only) or by downloading a raw GitHub Release asset.

This issue adds **on-demand binary provisioning**: keep the PyPI wheel tiny and pure-Python, and on first managed-mode use, download the matching `mlxcel` binary from the GitHub Release, verify its sha256, cache it, and run it. This is the `playwright install` model. Connect mode (talking to an already-running server via `base_url=`/`socket=`) never needs a local binary and is unaffected.

## Decision: do NOT bundle the binary in the wheel

Measured `v0.3.2` artifact sizes (zip; a wheel is itself a zip, so wheel size is roughly the same):

| Asset | Size |
|---|---|
| `mlxcel-macos-aarch64.zip` | 59 MB |
| `mlxcel-linux-aarch64-cuda13.zip` | 144 MB |
| `mlxcel-linux-x86_64-cuda13.zip` | 277 MB |

PyPI default storage limits are 100 MB per file and 10 GB per project (increases require filing a request at pypi/support after first uploading an under-limit release, then justifying it). So a Linux wheel (277 MB) exceeds the per-file limit outright, and bundling across versions and arches would burn the 10 GB project quota quickly. Linux binaries are also pinned to CUDA 13, and wheel tags cannot express a CUDA version, so a `manylinux` wheel would silently install on CUDA 12 or CPU-only hosts where the binary cannot run. Bundling fat wheels on PyPI is therefore the wrong fit. On-demand download keeps PyPI hosting only the small client and selects the correct artifact (including the CUDA variant) at runtime.

## Goals

- `pip install mlxcel` stays small and pure-Python (`py3-none-any`); no size or PyPI-limit concerns.
- First managed-mode use auto-provisions a version-matched binary on supported platforms, with integrity verification and caching, so subsequent runs are offline-fast.
- An explicitly installed binary (`binary=`, `MLXCEL_BIN`, or `mlxcel` on `PATH`, for example a brew install) is always preferred; auto-download is a fallback only when no binary is present.
- An explicit opt-out for environments that must not fetch, and an explicit pre-provision command for CI or air-gapped staging.
- Graceful, clear behavior when offline, on an unsupported platform, or on a CUDA mismatch.

## Non-goals

- No fat wheels on PyPI; no compiling the engine from source in the Python package.
- No automatic download at pip-install time. pip has no reliable post-install hook for wheels, so provisioning is lazy at first managed-mode use, or via an explicit command.
- No changes to the Rust core or to the release pipeline that produces the artifacts.
- No in-process embedding of the engine (that is Phase 2, the PyO3 binding).

## Design

### Artifact mapping

Map (OS, arch) to the release asset name (`mlxcel--[-cuda13].zip`; macOS carries no accelerator suffix, Metal is implicit):

- Darwin / arm64 -> `mlxcel-macos-aarch64.zip`
- Linux / x86_64 -> `mlxcel-linux-x86_64-cuda13.zip`
- Linux / aarch64 -> `mlxcel-linux-aarch64-cuda13.zip`
- anything else -> a clear unsupported-platform error pointing at connect mode and manual install.

For Linux, document the CUDA 13 assumption: if the host CUDA differs, connect mode or a manual install is the path. (A future CUDA-detection step could refine asset selection.)

### Which version to fetch

The client pins a target mlxcel release (a `_PINNED_RELEASE` constant), so a given `mlxcel` client release always fetches a known-good, sha256-verified binary rather than silently tracking "latest" (which could break compatibility). Allow an override via env (for example `MLXCEL_RELEASE`) for power users, and document the client-version to engine-version mapping in the package metadata and docs.

### Integrity

Download both the `.zip` and its `.sha256` sibling from the release and verify before extracting. Refuse to run an artifact that fails verification. Fail loudly on HTTP errors (no silent zero-byte download). For defense in depth, optionally also pin the expected sha256 inside the package, so a tampered release asset is caught even if its `.sha256` is swapped too.

### Cache location and layout

Extract the binary to a per-user cache directory via `platformdirs` (for example `/mlxcel/bin//mlxcel`), keyed by version so multiple versions coexist and upgrades stay clean. Set the executable bit. Reuse the cached binary on subsequent runs (no re-download). Provide a helper to locate and to clear the cache.

### Provisioning trigger and resolution order

Lazy provisioning. `ManagedServer._find_binary` resolution order becomes: `binary=` argument, then `MLXCEL_BIN`, then `mlxcel` on `PATH`, then the cached downloaded binary, then (if nothing is found and auto-download is enabled) download-now, otherwise the existing actionable error. Rationale: an intentionally installed binary (brew/PATH) wins; auto-download only kicks in when no binary exists, so this is purely additive to #407 and least-surprising. When a PATH binary's version differs from the client's pinned release, optionally log a one-line note.

Also expose explicit entry points so users can pre-provision without running a model:

- `mlxcel.download_binary(version=None, force=False) -> str`
- `python -m mlxcel install`

And an opt-out for locked-down environments: `auto_download=False` on the client, or `MLXCEL_NO_DOWNLOAD=1`.

### Network and UX

- First run logs a clear line (to the `mlxcel.server` logger / stderr): "downloading mlxcel (~59 MB) from ", with progress if feasible.
- Sensible timeouts and a couple of retries for transient GitHub failures.
- Offline with no cache and no installed binary -> `MlxcelServerError` with the manual-install hint (brew command plus the direct asset URL), not a hang.

## Implementation sketch

New module `mlxcel/_provision.py` (artifact-name mapping, version pin + env override, httpx download, sha256 verify, extract to the `platformdirs` cache, resolve). `_find_binary` gains the cached/download fallback and the opt-out. Add the `download_binary()` API and a `python -m mlxcel install` entry point. Dependencies: `httpx` is already a dep; add `platformdirs` (small) for the cache directory. Packaging is unchanged and stays `py3-none-any`.

## Tasks

- [ ] `_provision.py`: (OS, arch) -> asset-name mapping, pinned release + `MLXCEL_RELEASE` override, httpx download of `.zip` + `.sha256`, verification (release sha256 plus optional in-package pin), extract to a `platformdirs` cache keyed by version, set exec bit, resolve cached path.
- [ ] Wire the `_find_binary` fallback (order: `binary=` -> `MLXCEL_BIN` -> `PATH` -> cached -> download) with `auto_download` / `MLXCEL_NO_DOWNLOAD` opt-out.
- [ ] Explicit `mlxcel.download_binary()` API and `python -m mlxcel install` CLI for pre-provisioning (CI, air-gapped).
- [ ] Robust errors and progress logging: offline, HTTP failure, checksum mismatch, unsupported platform, Linux CUDA-mismatch note, each with a manual-install hint.
- [ ] Tests (binary-free in CI): mock the HTTP download with `httpx.MockTransport` serving a tiny fake "binary" plus a matching sha256, then assert verify-pass, verify-fail rejection, cache reuse (no second download), resolution order (an installed binary wins over download), the opt-out, and the unsupported-platform error. Keep the real-network real-Release test marked and opt-in.
- [ ] Add `platformdirs` to deps; add a cache locate/clear helper.
- [ ] Docs (`docs/python-client.md`, English then Korean): the install matrix (`pip install mlxcel` then auto-provision on first managed use, `python -m mlxcel install` to pre-fetch, the opt-out env, the cache location, brew and connect-mode alternatives, the Linux CUDA 13 note).

## Acceptance criteria

- `pip install mlxcel` produces a small pure-Python wheel with no size or PyPI-limit issue.
- On macOS arm64 with no preinstalled binary, the first `mlxcel.LLM("")` (or `python -m mlxcel install`) downloads the pinned `mlxcel`, verifies its sha256, caches it, and managed mode runs; a second run reuses the cache with no re-download.
- A checksum mismatch or HTTP failure aborts with a clear error and a manual-install hint, and never runs an unverified binary.
- An installed binary (`binary=` / `MLXCEL_BIN` / `PATH`) is used in preference to downloading; `MLXCEL_NO_DOWNLOAD=1` disables auto-download.
- Offline with no cache and no binary yields an actionable error, not a hang.
- Docs cover the matrix in English and Korean. No Rust or release-pipeline changes.

## Relation to #407 and Phase 2

This is additive to #407's external-binary discovery (download is a new fallback step, installed binaries still win). It is independent of Phase 2 (the in-process PyO3 binding), which would embed the engine for in-process inference rather than spawning the server executable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.