AcademySoftwareFoundation / AcademySoftwareFoundation/rez
[Feature] Support non-rez package definition files (eg setup.py)
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 369
- Avg merge
- 12d 3h
- Merged PRs (30d)
- 5
Description
### Goal
Eliminate vendor lock-in.
### Motivation
In order for a Python project to be both PyPI and Rez compatible, you need two project definition files, a `setup.py` and `package.py`. Likewise for a project to compatible with conan or vcpkg or yum or any other package manager out there, you need 1 more package definition.
With `rez pip --install`, we've demonstrated that Rez as a build-system can be entirely stripped out and delegated to a third-party. That the workflow is sound and that it avoids the need to write a second-class package definition file for each package in order to benefit from what makes Rez strong - dynamically creating environments from a self-curated repository of packages.
I'm integrating Rez with a studio at the moment, and one of the things we'll have to do is replace our existing (internal) package manager file-format (not unlike Rez) with Rez's `package.py` file. If at any point in the future we decide to move to the next great thing, history will repeat itself. If instead packages had been written for PyPI and VcPkg, our eggs would not all have been in the same basket.
### Implementation
| Before | After
|:--|:---
|
| 
By removing the build-system aspect from Rez, we can instead focus on bridging the gap between external build systems, allowing both existing projects from working natively and instantly with Rez - such as those already written for Conan or Vcpkg - and make any build definition written for Rez be natively dual-compatible with an external service, such as PyPI or yum.
In practice, the workflow could look like this.
**Before**
1. Author your project
2. Write your `package.py` for Rez
3. Write your `setup.py` for PyPI
**After**
1. Author your project
2. Write your `setup.py` for Rez and PyPI
### What about package.py?
To clarify, there are two `package.py` files.
1. The one you author
2. The one released
(2) remains unchanged, making this idea compatible with all of your existing packages and repositories.
(1) however may no longer be necessary. Each external build-system would effectively generate (2). A `setup.py` covers most if not all requirements of a `package.py`, and being plain-Python means it can be extended to contain additional fields. Likewise for Conan or CMake definitions. Odds are the essence of a `package.py` can be reduced into additional fields of an existing package definition format.
### Integration
Akin to `rez pip`, Rez wraps sub-managers to provide additional infrastructure, such as what options and cmake modules to provide.
```bash
$ rez pip --install .
$ rez conan --install .
$ rez vcpkg --install .
```
The subcommand, e.g. `pip`, identifies the format the package definition file is in, and handled similar to `rez/pip.py`, although it should probably be delegated to a plug-in.
### Benefits
- [x] **Instant familiarity** A new user with an existing package library on PyPI and local repository is already able to start using Rez with little to no change. Likewise users familiar with any other (supported) build system
- [ ] **No vendor lock-in** Encourage new users to start using Rez without having to commit, and provide existing users with ease of mind that their investment into writing package definition files are safe and transferable to another system, e.g. everything was written for Conan.
- [ ] **Less maintenance** Rez is both a build-system and environment management system, the less there is to manage, the more focus it can get.
- [ ] **Cross-repository dependencies** Build a C++ project with dependencies coming straight out of conan, vcpk, pip and more. No need to convert the dozens of dependencies prior to making it Rez-compatible.
### Testcase
Can this work? `rez pip` has already proven that it can. With it, it's possible right now to stop writing `package.py` files for any Python project, and in addition have your project instantly compatible with PyPI. The same can be true for just about any established build system out there.
### Discussion
Even though your existing package repositories remain compatible, this is a big change to the fundamental nature of Rez. It would turn it into something other than what it is. Something stronger, more welcoming of new users and more forgiving to those that commit long-term. But it is still a big change.
Contributor guide
Research direction
Start by reading the existing `rez pip --install` workflow and `rez/pip.py`, then compare how `setup.py` is interpreted with the generated `package.py` described in the issue. The proposed work would need a decided plugin or subcommand design and a clear test plan covering external package definition formats and generated Rez packages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100