python-poetry / python-poetry/poetry
Multi-platform split-repo packages cause errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.3k
- Forks
- 2.5k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 30
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
Issue
Unlike pip, poetry seems to assume that wheels for different platforms of the same package
live in the same repository. This is more than a little surprising for those used to the pip behavior.
At the very least, this should be clearly documented, or -preferably- fixed.
Also note that one common use of private repos is exactly to host wheels for architectures
not produced by the original project. Surely, those "standard" wheels can be re-hosted in the
private repo, but that's not exactly convenient or resource efficient.
Below I include a pyproject.toml along with example runs in ARM64 and AMD64 containers
that illustrate the problem. Click on the corresponding text to expand the relevant section.
At the time of writing, the private repo only included the ARM64 version of the target package.
[This will change as soon as this bug report is submitted to work around this shortcoming.]
pyproject.toml
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "test"
version = "0.1.0"
description = "Multi-platform split-repo package"
authors = ["surprised"]
[[tool.poetry.source]]
name = "ray-wheels"
url = "https://mehes-kth.github.io/ray-wheels/simple"
[tool.poetry.dependencies]
python = "~3.8"
ray = "^1.6.0"
ARM64
user@host$ docker run --rm -it \
> --platform linux/arm64 \
> -e PIP_EXTRA_INDEX_URL=https://mehes-kth.github.io/ray-wheels/simple \
> -v $PWD/pyproject.toml:/pyproject.toml \
> python:3.8 bash
Unable to find image 'python:3.8' locally
3.8: Pulling from library/python
Digest: sha256:a0a734233420b17d9ab37125afc9d8217b75db153d55854ac6683e639f00a8e8
Status: Downloaded newer image for python:3.8
root@f39b82f3a485:/# pip download --verbose --no-deps ray==1.6.0
Looking in indexes: https://pypi.org/simple, https://mehes-kth.github.io/ray-wheels/simple
Collecting ray==1.6.0
Downloading https://mehes-kth.github.io/ray-wheels/simple/ray/ray-1.6.0-cp38-cp38-linux_aarch64.whl (24.5 MB)
|████████████████████████████████| 24.5 MB 8.1 MB/s
Saved /ray-1.6.0-cp38-cp38-linux_aarch64.whl
Successfully downloaded ray
root@f39b82f3a485:/# pip install poetry >& /dev/null
root@f39b82f3a485:/# poetry --version
Poetry version 1.1.8
root@f39b82f3a485:/# poetry install
Creating virtualenv test-il7asoJj-py3.8 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (10.1s)
Writing lock file
Package operations: 11 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing attrs (21.2.0)
• Installing click (8.0.1)
• Installing filelock (3.0.12)
• Installing grpcio (1.39.0)
• Installing msgpack (1.0.2)
• Installing numpy (1.21.2)
• Installing protobuf (3.17.3)
• Installing pyyaml (5.4.1)
• Installing redis (3.5.3)
• Installing ray (1.6.0)
root@f39b82f3a485:/#
exit
user@host$
AMD64
user@host$ docker run --rm -it \
> --platform linux/amd64 \
> -e PIP_EXTRA_INDEX_URL=https://mehes-kth.github.io/ray-wheels/simple \
> -v $PWD/pyproject.toml:/pyproject.toml \
> python:3.8 bash
Unable to find image 'python:3.8' locally
3.8: Pulling from library/python
Digest: sha256:a0a734233420b17d9ab37125afc9d8217b75db153d55854ac6683e639f00a8e8
Status: Downloaded newer image for python:3.8
root@7cb84f3b0417:/# pip download --verbose --no-deps ray==1.6.0
Looking in indexes: https://pypi.org/simple, https://mehes-kth.github.io/ray-wheels/simple
Collecting ray==1.6.0
Downloading ray-1.6.0-cp38-cp38-manylinux2014_x86_64.whl (49.3 MB)
|████████████████████████████████| 49.3 MB 14.9 MB/s
Saved /ray-1.6.0-cp38-cp38-manylinux2014_x86_64.whl
Successfully downloaded ray
root@7cb84f3b0417:/# pip install poetry >& /dev/null
root@7cb84f3b0417:/# poetry --version
Poetry version 1.1.8
root@7cb84f3b0417:/# poetry install
Creating virtualenv test-il7asoJj-py3.8 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (70.7s)
Writing lock file
Package operations: 11 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing attrs (21.2.0)
• Installing click (8.0.1)
• Installing filelock (3.0.12)
• Installing grpcio (1.39.0)
• Installing msgpack (1.0.2)
• Installing numpy (1.21.2)
• Installing protobuf (3.17.3)
• Installing pyyaml (5.4.1)
• Installing redis (3.5.3)
• Installing ray (1.6.0): Failed
RuntimeError
Unable to find installation candidates for ray (1.6.0)
at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:72 in choose_for
68│
69│ links.append(link)
70│
71│ if not links:
→ 72│ raise RuntimeError(
73│ "Unable to find installation candidates for {}".format(package)
74│ )
75│
76│ # Get the best link
root@7cb84f3b0417:/#
exit
user@host$
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported pyproject.toml and reproduce poetry install on the ARM64 and AMD64 containers, then inspect poetry/installation/chooser.py at the failing line 72. Compare the resolver and candidate-selection behavior with pip for the split-repository ray wheels. Done means the package installs on both platforms, or the limitation and workaround are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100