python-poetry / python-poetry/poetry

Python restricted dependency not working

Open
#6,895 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

  • Poetry version: 1.2.2

  • Python version: 3.9.13

  • OS version and name: macOS Catalina, Version 0.15.7

  • I am on the latest stable Poetry version, installed using a recommended method.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • I have consulted the FAQ and blog for any relevant entries or release notes.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Hi, I'm currently working on a multirepo context and I have a pyproject.toml file that has some other repos between its dependencies. I tried to recreate the situation with three empty repos lib_a, lib_b, lib_c where lib_a depends on the other two.
To reproduce the error just clone this repo: https://github.com/carlotardivo/poetry-issue-6895 and execute a poetry lock -vvv from lib_a root directory.

These are the three pyproject.toml files:

lib_a:

[tool.poetry]
name = "lib_a"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "liba"}]

[tool.poetry.dependencies]
python = ">=3.7.13,<3.10"
lib_b={path='../lib_b'}
lib_c={path='../lib_c', optional= true, python = ">=3.9,<3.10"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

lib_b

[tool.poetry]
name = "lib_b"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "libb"}]

[tool.poetry.dependencies]
python = ">=3.7.13,<3.10"
scipy=[
    {version='1.5.*', python='~3.7.13'},
    {version='1.8.0', python='>=3.8'}
]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

lib_c:

[tool.poetry]
name = "lib_c"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "libc"}]

[tool.poetry.dependencies]
python = ">=3.9,<3.10"
scipy = { version = "^1.7.0", python = ">=3.9,<3.10" }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Expected result

Basically what I'd expect, according to these docs (python restrcted dependencies) is to be able to create a lock file that allows for the creation of an environment with

  • lib_b (with scipy 1.5.*) when python ~3.7.13
  • lib_b (with scipy 1.8.0) and lib_c installed only when python >=3.9,<3.10

Actual result

The lock file generation breaks with the following error, that seems related to the fact that scipy 1.5 is looked also for python >=3.9,<3.10 (while only ^1.7.0 should in this python range):

(poetry-experiment-py3.9) bash-3.2$ poetry lock -vvv
Loading configuration file /<my-user-local-path>/Library/Preferences/pypoetry/config.toml
Loading configuration file /<my-user-local-path>/Library/Preferences/pypoetry/auth.toml
Using virtualenv: /<my-user-local-path>/Library/Caches/pypoetry/virtualenvs/poetry-experiment-7exzTMzg-py3.9
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: lib-a is 0.1.0
   1: derived: lib-a
   1: fact: lib-a depends on lib_b (0.1.0)
   1: fact: lib-a depends on lib_c (0.1.0)
   1: selecting lib-a (0.1.0)
   1: derived: lib_c (0.1.0) @ ../lib_c
   1: derived: lib_b (0.1.0) @ ../lib_b
   1: fact: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: selecting lib-c (0.1.0 /<my-user-local-path>/Desktop/poetry_experiment/lib_c)
   1: derived: scipy (>=1.7.0,<2.0.0)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /simple/scipy/ HTTP/1.1" 304 0
[filelock] Attempting to acquire lock 4437848368 on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Lock 4437848368 acquired on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Attempting to release lock 4437848368 on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Lock 4437848368 released on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
Source (PyPI): 10 packages found for scipy >=1.7.0,<2.0.0
   0: Duplicate dependencies for scipy
   0: Different requirements found for scipy (1.5.*) with markers python_full_version >= "3.7.13" and python_full_version < "3.8.0" and scipy (1.8.0) with markers python_version >= "3.8".
   1: Version solving took 0.233 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (>=1.5.0,<1.6.0)>}}).
   1: fact: lib-a is 0.1.0
   1: derived: lib-a
   1: fact: lib-a depends on lib_b (0.1.0)
   1: fact: lib-a depends on lib_c (0.1.0)
   1: selecting lib-a (0.1.0)
   1: derived: lib_c (0.1.0) @ ../lib_c
   1: derived: lib_b (0.1.0) @ ../lib_b
   1: fact: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: selecting lib-c (0.1.0 /<my-user-local-path>/Desktop/poetry_experiment/lib_c)
   1: derived: scipy (>=1.7.0,<2.0.0)
Source (PyPI): 10 packages found for scipy >=1.7.0,<2.0.0
   1: fact: lib-b (0.1.0) depends on scipy (1.5.*)
   1: conflict: lib-b (0.1.0) depends on scipy (1.5.*)
   1: derived: scipy (>=1.5.0,<1.6.0)
   1: conflict: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: ! not scipy (>=1.7.0,<2.0.0) is satisfied by scipy (>=1.5.0,<1.6.0)
   1: ! which is caused by "lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*)"
   1: ! thus: lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b
   1: ! lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b is satisfied by lib_b (0.1.0) @ ../lib_b
   1: ! which is caused by "lib-a depends on lib_b (0.1.0) @ ../lib_b"
   1: ! thus: lib-c is forbidden
   1: ! lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is satisfied by lib_c (0.1.0) @ ../lib_c
   1: ! which is caused by "lib-a depends on lib_c (0.1.0) @ ../lib_c"
   1: ! thus: version solving failed
   1: Version solving took 0.016 seconds.
   1: Tried 1 solutions.

  Stack trace:

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│ 
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:127 in solve
      125│             while next is not None:
      126│                 self._propagate(next)
    → 127│                 next = self._choose_package_version()
      128│ 
      129│             return self._result()

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:446 in _choose_package_version
      444│             package = locked
      445│ 
    → 446│         package = self._provider.complete_package(package)
      447│ 
      448│         conflict = False

  OverrideNeeded

  ({Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (>=1.5.0,<1.6.0)>}}, {Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (==1.8.0)>}})

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:764 in complete_package
      760│                     current_overrides.update({dependency_package: package_overrides})
      761│                     overrides.append(current_overrides)
      762│ 
      763│             if overrides:
    → 764│                 raise OverrideNeeded(*overrides)
      765│ 
      766│         # Modifying dependencies as needed
      767│         clean_dependencies = []
      768│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│ 
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:126 in solve
      124│             next: str | None = self._root.name
      125│             while next is not None:
    → 126│                 self._propagate(next)
      127│                 next = self._choose_package_version()
      128│ 

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:165 in _propagate
      163│                     # where that incompatibility will allow us to derive new assignments
      164│                     # that avoid the conflict.
    → 165│                     root_cause = self._resolve_conflict(incompatibility)
      166│ 
      167│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c depends on scipy (^1.7.0)
   and lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*), lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b.
  So, because lib-a depends on both lib_b (0.1.0) @ ../lib_b and lib_c (0.1.0) @ ../lib_c, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:364 in _resolve_conflict
      360│             )
      361│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      362│             self._log(f"! thus: {incompatibility}")
      363│ 
    → 364│         raise SolveFailure(incompatibility)
      365│ 
      366│     def _choose_package_version(self) -> str | None:
      367│         """
      368│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:115 in run
       113│             self._execute_operations = False
       114│ 
     → 115│         return self._do_install()
       116│ 
       117│     def dry_run(self, dry_run: bool = True) -> Installer:

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:248 in _do_install
       246│                 source_root=self._env.path.joinpath("src")
       247│             ):
     → 248│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       249│         else:
       250│             self._io.write_line("Installing dependencies from lock file")

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:73 in solve
        71│         with self._provider.progress():
        72│             start = time.time()
     →  73│             packages, depths = self._solve(use_latest=use_latest)
        74│             end = time.time()
        75│ 

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:157 in _solve
       155│             packages = result.packages
       156│         except OverrideNeeded as e:
     → 157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
       158│         except SolveFailure as e:
       159│             raise SolverProblemError(e)

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:116 in solve_in_compatibility_mode
       114│             )
       115│             self._provider.set_overrides(override)
     → 116│             _packages, _depths = self._solve(use_latest=use_latest)
       117│             for index, package in enumerate(_packages):
       118│                 if package not in packages:

  SolverProblemError

  Because lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c depends on scipy (^1.7.0)
   and lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*), lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b.
  So, because lib-a depends on both lib_b (0.1.0) @ ../lib_b and lib_c (0.1.0) @ ../lib_c, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:159 in _solve
      155│             packages = result.packages
      156│         except OverrideNeeded as e:
      157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      158│         except SolveFailure as e:
    → 159│             raise SolverProblemError(e)
      160│ 
      161│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      162│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      163│

I'm I missing something?

Thank you very much.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Clone the linked reproduction repository and run poetry lock -vvv from the lib_a root to confirm the failure. Start by reading poetry/puzzle/solver.py, poetry/mixology/version_solver.py, and poetry/puzzle/provider.py around the reported stack frames and override handling. Done means the example resolves its Python-restricted dependencies and the existing failure path is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.