Install packages from artifactory instead of conda inside the Met Office
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
### What problem does your feature request solve?
Currently we pull all of our dependencies from conda-forge directly.
While this currently works, it will soon stop working at the Met Office due to the [conda.anaconda.org domain being blocked (link is Met Office only)](https://metoffice.sharepoint.com/sites/TechSolDelAVDTeam/SitePages/Anaconda-URL-blocking.aspx). Instead we should pull our dependencies from the JFrog Artifactory mirror of conda-forge instead.
Per Met Office secure development policy we should be pulling from artifactory anyway, and this blocking of the URL (IIRC motivated by the Anaconda Inc. licencing shenanigans) just enforces that. It is also good from a responsible use of common resources, as by pulling from the mirror we are reducing our load on conda-forge's CDN.
### Describe the solution you'd like
There are two cases where we should be installing from artifactory:
1. When installing on a Met Office machine.
2. When installing in a GitHub Action for a repository in the MetOffice GitHub organisation.
However, as non-Met Office people won't have artifactory access we need to make sure they can still access it.
3. When installing on non-Met Office machines.
4. When installing in a GitHub Action for a fork outside the Met Office organisation.
Achieving this will require some logic in the install script/`make setup`. A reliable way to detect we are on a Met Office machine is to check the hostname.
```sh
hostname -f | grep -qF metoffice
```
For GitHub Actions we can instead check the organisation from the GitHub context:
```yaml
jobs:
example:
steps:
- Other steps...
- name: Create conda environment.
env:
REPO_OWNER: ${{ github.repository_owner }}
run: |
if [[ "${REPO_OWNER}" = "MetOffice" ]]; then
# Use artifactory mirror.
else
# Use conda-forge directly.
fi
```
Once we have checked the name, we can rewrite the URLs in the lock files with a quick `sed` command:
```sh
sed -i "s|conda.anaconda.org|metoffice.jfrog.io/metoffice/api/conda|" requirements/locks/*.txt
```
For local runs we will also want to restore the lock files back afterwards `git restore requirements/locks`, so as to avoid leaving an uncommitted difference. We can probably skip this in GitHub Actions, as we don't commit them back (except in the update action that doesn't use the lockfiles).
To start with we should implement the changes for the local setup, as this is what will be broken by the blocking of the anaconda.org domain.
Using artifactory for our GitHub Actions requires additional work from others in the Met Office (which I'm separately pursuing in https://github.com/MetOffice/action-setup-artifactory/issues/15), so we can probably leave that for a separate effort, as it will suffer no immediate breakage.
### Describe alternatives you've considered
We alternatively could provide documentation to manually install from the right source, however this would be complicated to follow, and would undoubtable lead to issues, especially as the blocking error isn't clear (its served a redirect to a HTML page, rather than a clear 4XX error).
A more palatable alternative would be to move our package management to [pixi](https://pixi.prefix.dev/). Pixi allows for a central configuration to include replacements for URLs, such as replacing conda-forge with a mirror, and this works with lock files too. However, pixi would be a reasonably big change to our packaging, and unlike conda it isn't installed by default on most HPC/scientific computing systems. Given installing it at the Met Office requires fiddling around with conda installing the pixi client, it would probably cause more trouble than it is worth, at least for now.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.