lablup / lablup/backend.ai

Our own PyPI repository to host Backend.AI-optimized Python packages

Open
#160 1 comment 0 reactions 2 assignees Claimed by @hephaex View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

We are now managing our own builds of TensorFlow and Alpine Linux version of precompiled Python pacakges.

Let's host them on a private PyPI repository so that we could simplify docker-build steps here.For instance, we could run a separate script to build those packages using containers and automatically upload them to the private PyPI repository. Then downloading/installing those packages inside the Dockerfiles will become much simpler than using multi-staged builds. (The big shortcoming of multi-staged builds is that it is difficult to control caches and intermediate image name/tags...)

Previously it was cumbersome and resource-consuming to host a private PyPI repository because we had to run a dedicated server. But now, there is a tool called [s3pypi](https://github.com/novemberfiveco/s3pypi) which allows us to build and host a PyPI repository with only S3 static website hosting + CloudFront.

Suggested repository URLs:

- `https://pypi.backend.ai/kernels/alpine/3.8/`
- `https://pypi.backend.ai/kernels/ubuntu/16.04/`
- `https://pypi.backend.ai/kernels/ubuntu/18.04/`
This could eliminate use of our "build.py" script and complicated build chains.

Also, @tlqaksqhr's https://github.com/lablup/backend.ai-packages could be hosted in this way as well, probably via:

- `https://pypi.backend.ai/dist/ubuntu/16.04/` or something similar.In this case the repository is for distributing Backend.AI itself.Specifically, we could also make a secret repository for enterprise customers.

JIRA Issue: BA-326

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.