AcademySoftwareFoundation / AcademySoftwareFoundation/rez

[Feature] Support non-rez package definition files (eg setup.py)

Open
#642 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
369
Avg merge
12d 3h
Merged PRs (30d)
5

Description

### Goal

Eliminate vendor lock-in.

### Motivation

In order for a Python project to be both PyPI and Rez compatible, you need two project definition files, a `setup.py` and `package.py`. Likewise for a project to compatible with conan or vcpkg or yum or any other package manager out there, you need 1 more package definition.

With `rez pip --install`, we've demonstrated that Rez as a build-system can be entirely stripped out and delegated to a third-party. That the workflow is sound and that it avoids the need to write a second-class package definition file for each package in order to benefit from what makes Rez strong - dynamically creating environments from a self-curated repository of packages.

I'm integrating Rez with a studio at the moment, and one of the things we'll have to do is replace our existing (internal) package manager file-format (not unlike Rez) with Rez's `package.py` file. If at any point in the future we decide to move to the next great thing, history will repeat itself. If instead packages had been written for PyPI and VcPkg, our eggs would not all have been in the same basket.

### Implementation

| Before | After
|:--|:---
| |

By removing the build-system aspect from Rez, we can instead focus on bridging the gap between external build systems, allowing both existing projects from working natively and instantly with Rez - such as those already written for Conan or Vcpkg - and make any build definition written for Rez be natively dual-compatible with an external service, such as PyPI or yum.

In practice, the workflow could look like this.

**Before**

1. Author your project
2. Write your `package.py` for Rez
3. Write your `setup.py` for PyPI

**After**

1. Author your project
2. Write your `setup.py` for Rez and PyPI

### What about package.py?

To clarify, there are two `package.py` files.

1. The one you author
2. The one released

(2) remains unchanged, making this idea compatible with all of your existing packages and repositories.

(1) however may no longer be necessary. Each external build-system would effectively generate (2). A `setup.py` covers most if not all requirements of a `package.py`, and being plain-Python means it can be extended to contain additional fields. Likewise for Conan or CMake definitions. Odds are the essence of a `package.py` can be reduced into additional fields of an existing package definition format.

### Integration

Akin to `rez pip`, Rez wraps sub-managers to provide additional infrastructure, such as what options and cmake modules to provide.

```bash
$ rez pip --install .
$ rez conan --install .
$ rez vcpkg --install .
```

The subcommand, e.g. `pip`, identifies the format the package definition file is in, and handled similar to `rez/pip.py`, although it should probably be delegated to a plug-in.

### Benefits

- [x] **Instant familiarity** A new user with an existing package library on PyPI and local repository is already able to start using Rez with little to no change. Likewise users familiar with any other (supported) build system
- [ ] **No vendor lock-in** Encourage new users to start using Rez without having to commit, and provide existing users with ease of mind that their investment into writing package definition files are safe and transferable to another system, e.g. everything was written for Conan.
- [ ] **Less maintenance** Rez is both a build-system and environment management system, the less there is to manage, the more focus it can get.
- [ ] **Cross-repository dependencies** Build a C++ project with dependencies coming straight out of conan, vcpk, pip and more. No need to convert the dozens of dependencies prior to making it Rez-compatible.

### Testcase

Can this work? `rez pip` has already proven that it can. With it, it's possible right now to stop writing `package.py` files for any Python project, and in addition have your project instantly compatible with PyPI. The same can be true for just about any established build system out there.

### Discussion

Even though your existing package repositories remain compatible, this is a big change to the fundamental nature of Rez. It would turn it into something other than what it is. Something stronger, more welcoming of new users and more forgiving to those that commit long-term. But it is still a big change.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing `rez pip --install` workflow and `rez/pip.py`, then compare how `setup.py` is interpreted with the generated `package.py` described in the issue. The proposed work would need a decided plugin or subcommand design and a clear test plan covering external package definition formats and generated Rez packages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.