MetOffice / MetOffice/LFRic-Atmosphere-Training

[Proposal] Introducing Pixi

Open
#162 3 comments 0 reactions 1 assignee Claimed by @ickc View on GitHub
Dependency Technical
Dominant language
Jupyter Notebook
Stars
11
Forks
6
Avg merge
5d 19h
Merged PRs (30d)
9

Description

We'll briefly describe why conda becomes essential in the Python ecosystem and why it may benefits this repo.

# PyPI vs. conda-forge

One of the dimension is where your packages come from, and how they are packaged. If you install from pip or uv, you get the packages from PyPI, where maintainers of the package upload them directly. The disadvantages of installing packages from PyPI include supply chain attack (there's is no layer between you and the maintainer, compromising a maintainer directly compromise you, c.f. #159), and poorly packaged packages. Here poor can mean: non-reproducible environments (package depends on external dependency or even compiler), improperly licensed software (missing LICENSE/LICENCE file). Now these may not be the case of the dependencies included in this repo, but this is a fact you need to verify per dependency, including their transient dependencies.

Now in conda, there are channels (more like APT repository if you're familiar with that), here I only focused on conda-forge channel, which is different from the main/default channel provided by Anaconda the company.

[conda-forge is a NumFOCUS project](https://numfocus.org/project/conda-forge), and is maintained by a group of maintainers independently. The infrastructure has imposed additional requirements so that every packages are built openly in CI with a set of requirements. For a package to exist on conda-forge, reproducibility (hermitic-ish build is used), licensing, etc. are guaranteed. And since it adds extra layers of maintainers (conda-forge maintainers and individual package maintainers), it puts more layers of defenses in a supply chain attack scenario.

Not to mention binaries build are generally better optimized because conda is actually language agnostics so it can bootstrap all the necessary compilers and libraries, but I digest.

# Conda environments is needed somewhere in the repo

E.g. in

```sh
./notebooks/README.md:18:This repository copy uses a conda-first setup for regridding practicals.
```

So it seems inevitable to include conda somewhere here, just a matter of where.

# pixi is like uv but for conda packages, with better reproducibility and a built in task runner

Like how a typical conda environment can include PyPI packages as well, pixi environment can includes PyPI packages by including uv in pixi (pixi solver actually uses uv within its binary to resolve PyPI dependency.) In this sense, the relationship of pixi vs. uv is just conda (as the package index) vs. PyPI. I.e. by default, pixi resolves packages using conda channels such as conda-forge.

Their design is similar, e.g. project centric rather than system wide.

The reproducibility aspect is better, generally because of the conda package ecosystem. E.g. the lockfile resolves per platform (see #157 for an example) and hence platform dependent problems can be detected earlier—you have more control and information on the environment you created.

It also includes a built-in task runner (see #157 for examples again) so that you don't need to rely on ad hoc Makefile or notes to remind you what can be done. Run `pixi task list` and see all tasks you can perform.

By the way, `pixi` is created by the same developer who creates `mamba`, the drop-in replacement of `conda` in the conda-forge community.

# Proposal: adding pixi as an optional developer experience

#157 serves as the proposal. It is not proposing to replace the current developer experience, pip and uv can continue to work. Adding it to the CI is an additional bonus that you'll know your workflow works regardless how your environment is built (from PyPI or conda-forge). It also serves as a prove that it is setup correctly even if others (reviewers) might not want / be able to use pixi. Once #157 is accepted, more thoughts can be given to see how various different conda environments currently defined as user instructions to be integrated together to provide a more uniform user experience. I.e. current proposal gears more towards on DX but it can also benefits UX once the foundation is laid.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.