conda-forge / conda-forge/conda-forge.github.io

RFC: conda-forge epochs for solver accuracy, speed & debuggability?

Đang mở
#1,867 21 bình luận 1 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
JavaScript
Star
170
Fork
320
Merge trung bình
2 ngày 10 giờ
Pull request đã merge (30 ngày)
5

Mô tả

Conda and mamba's solver take into account the entirety of packages ever published when trying to resolve an environment (with some accelerations, i.e. checking first if things are resolvable with `repodata_current.json`).

This can lead sometimes lead the solver astray and force it into very weird contortions, where very old packages are picked just because they _seemingly_ satisfy the constraints (though realistically, this is almost always an error in our metadata). There are many examples of this, here's a few that came up recently:
* #1528
* #1597
* https://github.com/pytorch/vision/issues/4665

While this definitely also has some advantages (less rebuilds, old packages stay installable), this also can run into inevitable problems where old packages haven't been rebuilt for modern dependencies (e.g. no run-exports), not aware of unknown-at-the-time ABI breaks, noarch vs. yesarch, etc.

So it would be nice to give users a way to enforce an option that says "I only want comparatively recent packages" or, in other words, "please don't do unexpected/unintended/crazy things while trying to resolve my environment".

I was thinking about how this could be done in a way that wouldn't require constant rebuilds (i.e. say, if a "conda-forge epoch" were to be defined as equal to a calendar year, nothing would be installable in January until all common packages have been rebuilt).

My current idea looks as follows:
* There's an empty metapackage `__conda-forge-epoch` that gets built every day (or week, or month), and versioned accordingly, i.e. `2022.12.19`.
* All outputs gain an automatic run-constraint
```
run_constrained:
{% set epoch = datetime.date.today().strftime('%Y.%m.%d') %}
- __conda-forge-epoch <={{ epoch }}
```
* note the `<=`, which is the other way around from e.g. our usual run-exports.
* implementing this (without having to modify every recipe) probably needs support from conda-build, but for now I'm assuming this is possible.
* By default, `__conda-forge-epoch` does not get installed, and therefore the constraints don't get triggered.
* This also means we wouldn't have to rebuild stuff more often than we already do, as the proposed default is effectively the same as the status quo.
* In other words, there are no hard "epoch breaks" (like we had once upon a time for going from the old compilers to the new ones).
* If a user wants to make avoid certain solver errors, or simply enforce recent builds, they can add `__conda-forge-epoch>=yyyy.mm.dd` to their environment specs (_now_ we have the `>=`). This would force the solver to only take into account packages built after that date.
* Perhaps even more importantly, _**it would allow users (& conda-forge members) to more easily debug**_ solver errors, by forcing the solver to only consider a more recent subset of packages, without getting lost in the weeds of the past.

I think just the debugging capabilities of this would make this worth considering, but maybe I'm just not very good at debugging resolver errors. 😅

Would be interested to hear people's thoughts.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.