conda-forge / conda-forge/conda-forge.github.io

Define channels better and document that

Aperta
#883 6 commenti 2 reazioni 0 assegnatari Vedi su GitHub
Docs
Lingua principale
JavaScript
Stelle
170
Fork
320
Merge medio
2g 10h
PR unite (30g)
5

Descrizione

On [request](https://github.com/conda-forge/pytorch-cpu-feedstock/issues/7#issuecomment-537024577) of @jakirkham, following up on the "channels" topic here. One of my questions from that discussion: _Does everything need to be in conda-forge? If so, what's the point of channels? If not, why can't we have cross-channel dependencies?_

So: what are channels, why do they exist, and what is the purpose of `defaults`, `conda-forge` and other channels?

**What is a channel?**
https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html says:
1. Conda packages are downloaded from remote channels, which are URLs to directories containing conda packages
2. Conda-forge is a community channel made up of thousands of contributors. Conda-forge itself is analogous to PyPI but with a unified, automated build infrastructure and more peer review of recipes.

@msarahan says:
"Channels are the notion of spaces where a coherent team dictates behavior. That team ideally uses a consistent toolchain across all packages in that channel. Package names are consistent within that channel. It is a technical solution to an arguably social problem - lining up practices, versions, and toolchains."

So there's some technical and some social aspects here that are relevant. The above feels incomplete though (if it's just URLs, what's the problem with having many; if it's consistency, why not just use a linter/checker or integration tests).

Actionable items:
- [ ] Update conda docs with a more complete answer aimed at conda users.
- [ ] Create an answer aimed at package authors and other potential contributors (in the conda-forge docs?).

**Why do we have defaults, conda-forge, and other channels?**

@msarahan says:
"Keep in mind, they are all targeting different runtime environments. The number of builds can't be any less than the number of different runtime environments."

I don't think that's the real answer, or correct. It's not true on Windows and macOS, and not really true on Linux either since `defaults` and `conda-forge` are _mostly_ compatible.

@isuruf says: "I guess you could see conda-forge and defaults as big package repositories like ubuntu and debian and pytorch as a ppa. .... We need channels like pytorch to be complementary to conda-forge. If so, we can easily share recipes and the maintenance burden will not be much."

I'll point out that it's about more than just recipes. E.g. PyTorch has a very large set of build/CI machines, it requires long, multi-stage build and test phases on GPU-enabled machines. If you copy the recipe to `conda-forge`, you still can't build it today. You don't want to do that copy: much better would be to verify that PyTorch builds are compatible, and then take over the built artifacts.

@bgruening says (to "what's the point of other channels"):
"For unstable stuff, for training, for beta-version, for people that don't like to play with communities or don't recognize the complexity of integration."

@h-vetinari says:
"It would be interesting to see how some "cross-compatible" channels would look like (or how that could ever be enforced in a way that gets the blessing of conda/conda-forge), but while it is mostly a social convention (as @rgommers mentions), there is a big impact of channels for corporate environments, where the rest of the internet is usually behind a proxy. Getting anything other than the main channels + conda-forge past IT / sysadmins / etc. is a hassle, both procedurally and technically, so every channel has a substantial incremental cost, while conda-forge just works (after the initial setup)."

Multiple people say: "less channels is always better".

To answer this question, let's say (as a thought experiment) we just move the few missing bits from `defaults` into `conda-forge`. We've got all packages, `conda-forge` works on the same Window/macOS/Linux platforms, so why not? Less confusion for users, less channels to deal with, win-win right?
What can go wrong here (leaving aside the issue of transitioning)?
- Control: `defaults` is controlled by Anaconda, they may have agreements with clients to do things a certain way (one close to home for me: modifying NumPy to insert Intel fft/random/linalg code)
- Quality assurance strategy is different: `defaults` probably has much more integration testing. In `conda-forge`, NumPy 1.17 is available straight away, in `defaults` it took two months, presumably because it broke some integration tests.
- Licenses needed: `defaults` may, e.g., contain packages built with commercial compilers that for some reason cannot be made available for `conda-forge`
- Hardware needed: GPU machines, very long builds, etc. cannot run on public free(-ish) CI. Paying the hardware setup and maintenance bill for `conda-forge` may be tricky.
- Toolchain incompatibilities: seems solvable, but there are some minor differences left I think. Unclear in how far there's a good reason for those.
- Targeting different OSes: no incompatibilities here today I think, but there could be (e.g. old Linux distros on HPC clusters, vs. latest-and-greatest with optimal performance)

So a channel seems much more a social than a technical concept. Also it can be about community building and scalability of organizing. E.g. Bioconda is separate from `conda-forge` not because of any of the above, but because it's easier to organize a team that way and because there may be bio-tools-specific issues that are easier to deal with in a separate group.
Large companies that produce tools and libraries (e.g. for PyTorch, RAPIDS, Tensorflow, CUDA, MXnet, Chainer, and so on) may also want separate channels (or GitHub orgs); if they spend millions on developing those tools and a large fraction of their users are installing with conda, then they will likely want to ensure releases are fast, things are built a certain way, etc. This can be achieved in multiple ways. It starts from: yes there can be real reasons to have multiple channels, or multiple GitHub orgs feeding packages into the same channel.

Actionable items:
- [ ] put in the conda docs why `defaults` exists, platforms it targets, etc.
- [ ] same for `conda-forge`, in the conda-forge docs
- [ ] put a clear recommendation into the docs for end users on when/why/how to mix channels
- [ ] put in the conda-forge docs how to create an organization like conda-forge that is complementary (e.g. like `bioconda`)
- [ ] think about how to bridge channels; separate control over a channel or GitHub org from what is end user visible.

And a more abstract meta comment: please think about requirements that follow from end user, library author, and IT admin perspectives and needs when discussing this (maybe I'm missing roles, the point is: _external_ stakeholders). Requirements come first; what are we trying to achieve? If the first thought to any of this is "well, but `glibc` or C++ ABI or ...", stop right there. Those are implementation details. Very important ones, and ones that can put constraints on requirements. But it's not interesting to a user, nor to people authoring libraries (not even when they want to package their library - only when there's an issue, then it becomes relevant).

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.