Proposal: discover script content from pip packages, not just ~/MLC/repos git clones

Proposal: discover script content from pip packages, not just ~/MLC/repos git clones

Today, script content (mlperf-automations's 377+ script/<alias>/ directories) only reaches mlcflow via mlc pull repo — a git clone into ~/MLC/repos/. This proposes a second, additive way to get content: pip-installed packages that self-advertise via a Python entry point, discovered generically (no hardcoded package names, forks included).

The approach

A content package (mlc-scripts, or a third-party fork) declares itself in its own pyproject.toml:

[project.entry-points."mlc.script_packages"]
mlperf-automations = "mlc_scripts_content"

mlcflow enumerates mlc.script_packages entry points on startup — a fast, local, no-network metadata read — and merges each into the same repo/index machinery git-cloned repos already use. Aliasing is by PyPI distribution name, not the self-chosen entry-point name, so a fork republished under a different distribution name coexists safely even if it reuses the same entry-point name.

What it looks like in practice

Installing is enough — no mlc pull repo step needed:

And running a script doesn't care where it came from — git-cloned and pip-installed sources are interchangeable, including cross-source dependencies:

Source precedence: pip first, ~/MLC/repos as fallback, flippable by env var

The Phase 1 prototype (mlcflow#274) currently does the opposite of the intended default: pip-sourced repos are prepended to the repo list, but repos processed later win uid conflicts, so a git clone in ~/MLC/repos silently overrides a pip-installed package today. That was fine for a prototype proving the mechanism, but it's not the intended long-term default.

Proposed default: pip-installed packages are checked first for a given script; ~/MLC/repos git clones are only consulted as a fallback when no installed package provides that script. Rationale — a pip install/pip install --upgrade is pinned and reproducible; a git clone in ~/MLC/repos can drift silently (stale git pull, local edits) without the user noticing. Pip-first makes the more trustworthy, more explicit source win by default.

Escape hatch: an env var reverses the order for anyone who needs the old behavior (local script development against a git checkout, debugging, CI pinned to a specific commit) without recompiling or reinstalling anything:

export MLC_SCRIPT_SOURCE_PRIORITY=git   # default is "pip"

This is a per-script fallback, not "pick one source and ignore the other": if the priority source doesn't have a given script (or dependency), mlcflow still finds it in the other source. Cross-source dependency resolution (a git-cloned script depending on a pip-sourced one, or vice versa) is unaffected by which order they're checked in — it already works either way per the script-action-flow diagram above.

Scope of this issue

This tracks the longer-term decision: whether/when to migrate real script content (the full 377-script tree) to a properly packaged pip distribution, along with the source-precedence default and env-var override described above. Phase 1 (mlcflow#274) is a prototype proving the discovery mechanism only — it does not touch the real script tree or automation/, and does not yet implement the pip-first default or the env var; that's follow-up work once this proposal is agreed.

Full design detail (including the fork-disambiguation rationale and consequences to plan for — read-only pip-sourced repos, "latest" becoming an explicit pip install --upgrade, packaging prerequisites) is in the attached slide deck; happy to share the full deck if useful.

Start by reading the Phase 1 prototype in mlcflow#274 and the package declaration in pyproject.toml. Then compare its discovery behavior with the script/ tree and automation/ scope described here. Done means the migration decision, pip-first precedence, and MLC_SCRIPT_SOURCE_PRIORITY override are agreed and documented.