python-poetry / python-poetry/poetry

When adding a package, extras in a set get reset to the default import

Open
#10,497 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

Description

Since I use PyTorch, I have a configuration for extras in a set that enable my users to select on which driver by the extra.

Click to expand the pyproject.toml file 
[project]
# classic stuff
dynamic = ["classifiers"]
dependencies = [  # This is used to generate metadata.
    "torch (==2.6.0+cpu); extra != 'cuda' and extra != 'cuda124' and extra != 'cuda118'",
    "torchvision (==0.21.0+cpu); extra != 'cuda' and extra != 'cuda124' and extra != 'cuda118'",
]

[project.optional-dependencies]
cuda = [
    "torch (==2.6.0+cu126)",
    "torchvision (==0.21.0+cu126)",
]
cuda124 = [
    "torch (==2.6.0+cu124)",
    "torchvision (==0.21.0+cu124)",
]
cuda118 = [
    "torch (==2.6.0+cu118)",
    "torchvision (==0.21.0+cu118)",
]

It causes two issues:

  • When installing from the wheel generated by poetry build without extras, the "default flavor", I don't get torch or torchvision installed
  • When I develop my software and add a dependency, for example poetry add pytimeparse, poetry reverts me back to the "default flavor" of torch (2.6.0+cpu), even though I installed with -E cuda (when producing the logs for this issue I found the issue also arises from poetry remove pytimeparse)
Workarounds

For the first issue, I told my end-users to run

poetry add --source my-configured-source my-project
poetry add "torch==2.6.0"
poetry add "torchvision==0.21.0"

When they don't have CUDA available

For the second one, I just have to continuously run poetry install ... whenever I add a new dependency

Poetry Installation Method

pipx

Operating System

Windows 10

Poetry Version

2.1.1

Poetry Configuration
cache-dir = "C:\\Users\\me\\AppData\\Local\\pypoetry\\Cache"
data-dir = "C:\\Users\\me\\AppData\\Roaming\\pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = false
keyring.enabled = true
python.installation-dir = "{data-dir}\\python"  # C:\Users\me\AppData\Roaming\pypoetry\python
repositories.pypi1.url = "https://pypi.org/simple/"
repositories.pypi2.url = "https://pypi.org/simple/"
repositories.pypi3.url = "https://pypi.org/simple/"
repositories.pytorch-cpu.url = "https://download.pytorch.org/whl/cpu"
repositories.pytorch-cuda118.url = "https://download.pytorch.org/whl/cu118"
repositories.pytorch-cuda124.url = "https://download.pytorch.org/whl/cu124"
repositories.pytorch-cuda126.url = "https://download.pytorch.org/whl/cu126"
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\alexandre.schoepp\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false
Poetry Runtime Logs
poetry-runtime.log 
Loading configuration file C:\Users\me\AppData\Roaming\pypoetry\config.toml
Loading configuration file C:\Users\me\AppData\Roaming\pypoetry\auth.toml
Loading configuration file C:\Users\me\Documents\Code\my-project\poetry.toml
Adding repository pytorch-cpu (https://download.pytorch.org/whl/cpu) and setting it as explicit
Adding repository pytorch-cuda126 (https://download.pytorch.org/whl/cu126) and setting it as explicit
Adding repository pytorch-cuda124 (https://download.pytorch.org/whl/cu124) and setting it as explicit
Adding repository pytorch-cuda118 (https://download.pytorch.org/whl/cu118) and setting it as explicit
Using virtualenv: C:\Users\me\AppData\Local\pypoetry\Cache\virtualenvs\my-project-k_pcpb9K-py3.12
Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[win32ctypes:win32ctypes.core.ctypes] Loaded ctypes backend
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'Windows WinVaultKeyring'
Available
Multiple source configurations found for pypi.org - pypi1, pypi2, pypi3, PyPI
Creating new session for pypi.org
Multiple source configurations found for pypi.org - pypi1, pypi2, pypi3, PyPI
Source (PyPI): 9 packages found for pytimeparse *
Using version ^1.1.8 for pytimeparse

Updating dependencies
Resolving dependencies...
 1: fact: my-project is 1.2.3
 1: derived: my-project
 0: Duplicate dependencies for torch
 0: Different requirements found for torch (2.6.0+cpu) with markers extra != "cuda118" and extra != "cuda124" and extra != "cuda", torch (<empty>) with markers (extra == "cuda" or extra == "cuda124" or extra == "cuda118") and (extra != "cuda" or extra == "cuda118" or extra == "cuda124") and (extra == "cuda" or extra == "cuda118" or extra != "cuda124") and (extra == "cuda" or extra != "cuda118" or extra == "cuda124"), torch (2.6.0+cu126) with markers extra == "cuda" and extra != "cuda118" and extra != "cuda124" and (extra == "cuda" or extra == "cuda118" or extra != "cuda124") and (extra == "cuda" or extra != "cuda118" or extra == "cuda124"), torch (2.6.0+cu124) with markers extra == "cuda124" and extra != "cuda" and extra != "cuda118" and (extra != "cuda" or extra == "cuda118" or extra == "cuda124") and (extra == "cuda" or extra != "cuda118" or extra == "cuda124") and torch (2.6.0+cu118) with markers extra == "cuda118" and extra != "cuda" and extra != "cuda124" and (extra != "cuda" or extra == "cuda118" or extra == "cuda124") and (extra == "cuda" or extra == "cuda118" or extra != "cuda124").
 1: Version solving took 0.003 seconds.
 1: Tried 1 solutions.
 0: Retrying dependency resolution with the following overrides ({Package('my-project', '1.2.3'): {'torch': <Dependency torch (==2.6.0+cpu)>}}).
 1: fact: my-project is 1.2.3
 1: derived: my-project
 ...

And the logs continue with installing the CPU version of torch

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with the shown pyproject.toml using poetry build, poetry add pytimeparse, and poetry remove pytimeparse, then compare the dependency-resolution logs for the extra markers and selected torch versions. Done means adding or removing a dependency no longer resets the selected extra, and building without extras produces the documented default dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.