Distribution story for BuildStream

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/332)
In GitLab by [[Gitlab user @tristanvb]](https://gitlab.com/tristanvb) on Mar 31, 2018, 10:18

We would like to move away from `pip` and replace this with a more distro friendly build system, possibly `meson`, for the purposes of installing BuildStream on a host, such that the leg work for a distro is reduced, and the reliability of BuildStream itself remains uncompromised.

## Problem statement

BuildStream currently uses a combination of manual steps and `pip` for installation purposes, but this is not ultimately a suitable approach for distributing BuildStream, as we require more deep integration with the underlying system, for some reasons:

* We depend on a number of bindings and not pure python dependencies, meaning we need separate manual steps and we cannot live in clean separation inside a *virtual env* at all because of this.
* We need to install data outside of `/usr/lib/python3/`, which is currently supported, but the `data_files` setuptools approach is in a deprecated state of flux which I have noted upstream pypi developers to be in disagreement about, I have been told to not use `data_files`, without any clear explanation of what I should use instead to install things on the host outside of python owned directories. Files we currently need to install include:
* Bash completion helper (we should be relying on a system installed `bash-completions.pc` file to discover the appropriate place to install our bash completion helper on the target host)
* Man pages, which should probably also use some host system introspection to figure out the appropriate installation directory, usually in `/usr/share/man`
* We should also improve our plugin story such that we no longer rely on `pip` for this, what I am hoping for is:
* BuildStream expects to find plugin packages in `${prefix}/share/buildstream/plugins`
* BuildStream installs a `buildstream.pc` pkg-config file, informing plugin packages where they should install
* Plugins use the BuildStream installed pkg-config file to discover where they should be installing to

## Complication of versions

In order to move to a more distro friendly installation model, we need to figure out a safe way of bringing in our pure python dependencies, which is tricky.

There is a cultural difference in the python community and the distro community:
* A great many python developers *expect* to be using `pip` for their distribution purposes, which allows python modules to depend on very strict versions because `pip` will install them in such a way that any version of a python dependency can be parallel installed.
* The distro community on the other hand usually carries with it a philosophy that a single computer should only need a single version of a given library at a time, as such the distro community expects installed modules to retain full API compatibility, and change the name of the shared library at a time when API compatibility breaks.

For BuildStream, we currently do not *"pin"* the exact versions of all of our pure python dependencies, but we certainly should be doing this at least in stable releases for reliability purposes - we cannot trust that pure python libraries adhere to the same standard of quality, because `pip` does not make this necessary for them.

## Plausible solutions

This problem needs a solution before we can think about dropping `pip` and moving to distribution model more appropriate for BuildStream. Some possible solutions I can think of include:
* Leveraging `pip` during the packaging phase, where we turn the BuildStream git checkout into a consumable tarball
* In this scenario, I would expect that we use `pip` to download the dependencies, and that we ensure that our `make install` step installs BuildStream in such a way that only the pure python dependencies which we've *bundled* into our tarball are ever used by BuildStream, and not any random version that a distro might have considered to be correct.
* Using submodules in the BuildStream repository to include the upstream dependencies in the BuildStream build directly

No source files or tests are named. Start by mapping the current pip, setuptools, and make install flow, then determine how host data, man pages, bash completion, plugins, and pure Python dependencies are handled; done requires an agreed distribution model that addresses these installation and versioning concerns.

Written by the indexing model from the issue text.