Our own PyPI repository to host Backend.AI-optimized Python packages
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
We are now managing our own builds of TensorFlow and Alpine Linux version of precompiled Python pacakges.
Let's host them on a private PyPI repository so that we could simplify docker-build steps here.For instance, we could run a separate script to build those packages using containers and automatically upload them to the private PyPI repository. Then downloading/installing those packages inside the Dockerfiles will become much simpler than using multi-staged builds. (The big shortcoming of multi-staged builds is that it is difficult to control caches and intermediate image name/tags...)
Previously it was cumbersome and resource-consuming to host a private PyPI repository because we had to run a dedicated server. But now, there is a tool called [s3pypi](https://github.com/novemberfiveco/s3pypi) which allows us to build and host a PyPI repository with only S3 static website hosting + CloudFront.
Suggested repository URLs:
- `https://pypi.backend.ai/kernels/alpine/3.8/`
- `https://pypi.backend.ai/kernels/ubuntu/16.04/`
- `https://pypi.backend.ai/kernels/ubuntu/18.04/`
This could eliminate use of our "build.py" script and complicated build chains.
Also, @tlqaksqhr's https://github.com/lablup/backend.ai-packages could be hosted in this way as well, probably via:
- `https://pypi.backend.ai/dist/ubuntu/16.04/` or something similar.In this case the repository is for distributing Backend.AI itself.Specifically, we could also make a secret repository for enterprise customers.
JIRA Issue: BA-326
Contributor guide
Assessment
This issue has not been assessed yet.