coin-or / coin-or/CyLP

Installing cylp in a robust, portable, neat way

Open
#57 10 comments 7 reactions 1 assignee Claimed by @tkralphs View on GitHub
enhancement
Dominant language
JetBrains MPS
Stars
192
Forks
70
PR merge metrics
No merged PRs in 30d

Description

This is following on from Issue #47
(I can't figure out how to reopen that issue)

# Intro

I want to use cylp on a system for which pip cannot find precompiled binaries. `pip install cylp` should still work. It should compile things for me, but it does not.

As a user, I expect to be able to install cylp **solely** by doing

pip install cylp

Requiring more steps than that is not ideal. Requiring users to manually download and manually compile many dependencies is not user friendly, nor is it robust.

Furthermore, I expect that using `pip install cylp` with a virtual environment will install cylp and all dependencies into that environment. For context, I'm deploying my code into AWS Lambda functions. That involves zipping up a virtual environment. For every library except cylp, I can just use pip, and zip up the directory containing the virtual environment.

I understand that packaging binaries in python libraries neatly is non-trivial. However it is certainly possible. Numpy and pandas use C files too, and they work seamlessly. I realise I'm asking for a lot by requesting that `pip` work seamlessly. I realise that this library and it's dependencies are probably written by a one or two people in their spare time. I'm not trying to criticise. I'm trying to help. That's why I've included published 2 scripts which took me several weeks to write. [This is what it takes](https://github.com/matt-tenergy/cylp-makescript) to install cylp in a robust, reliable and reproducible way (on machines for which pip cannot find pre-compiled binaries).

This process involved me figuring out the extensive dependency tree (including dependencies which pip does not know about), and the fragile version incompatibilities (e.g. did you know that there are 2 different versions of numpy both numbered 1.12.1?), and then manually downloading the source code and compiling each of those dependencies in their own custom virtual environment, extracting `.whl` files from one virtual env and installing it in the next. In short, it is a nightmare.

# The scripts

I've split up this script into 2 steps, because the first script takes a very long time, so you don't want to run it every time you rebuild your virtual environment.

* [`compile.sh`](https://github.com/matt-tenergy/cylp-makescript/blob/master/compile.sh) compiles all the libraries into `.whl` files
* [`makeScript.sh`](https://github.com/matt-tenergy/cylp-makescript/blob/master/makeScript.sh) script takes the `.whl` files compiled by `compile.sh`, and installs them into a virtual env, in such a way that pip is aware of (almost) all dependencies simultaneously

# Details and Outstanding issues

* This script needs to modify the `LD_LIBRARY_PATH` OS variable. That's messy and crude, and means this script can't easily go into a normal `setup.py` file.
* This script works on my machine, which is an EC2 instance running Amazon's version of Redhat. When I try running it on a fresh machine, I still get some issues. I'm just posting what I have anyway.

# Conclusion

I believe a user-friendly installation process is by far the most important issue facing this library, because the current process makes it near impossible to use.

I'm not sure whether the cause of all these issues is cylp itself, or rather CBC and SCS. I'm posting here first to see what you say.

What’s the best way to add these scripts to setup.py, so that pip does all this for the user? To restate my goal: I want to be able to install cylp and its dependencies into a self-contained virtual environment using a single pip command.

Or failing that, I just hope that other users struggling with the installation process can copy and paste my work here to save themselves weeks of hair pulling and confusion.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.