python-poetry / python-poetry/poetry

group-specific updates to extras work: unintended, or undocumented?

Open
#8,581 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/docs status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

  • I have searched the issues of this repo and believe that this is not a duplicate.

Issue

When looking up the docs on how to define extras, the final box says in very clear words, that you can only define the packages that you can list in a [tool.poetry.extras] entry within the [tool.poetry.dependencies]/[tool.poetry.dependencies.main] section:

The dependencies specified for each extra must already be defined as project dependencies.
Dependencies listed in dependency groups cannot be specified as extras.

This is not 100% true, I can re-define dependencies as optional in a group, and if I install that group plus the extra, poetry will consider the dependency as it is defined in the group.

For example, I have an extra snowflake which installs snowflake-connector-python with its pandas extra. During development, I'd like to install its secure-local-storage extra as well. I can express this with

[tool.poetry]
name = "group-extras"
version = "0.1.0"
description = ""
authors = ["Arne <me@arne.com>"]
readme = "README.md"
packages = [{include = "group_extras"}]

[tool.poetry.dependencies]
python = "^3.10"
snowflake-connector-python = {version = ">=3.2.1", optional = true, extras = ["pandas"]}

[tool.poetry.group.dev.dependencies]
snowflake-connector-python = {version = "*", optional = true, extras = ["secure-local-storage"]}

[tool.poetry.extras]
snowflake = ["snowflake-connector-python"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

And it will work just fine.

I expected a syntax error due to the option=true in a non-main group, but it resolves

$ poetry lock
Updating dependencies
Resolving dependencies... [time passes]
Writing lock file

This installs snowflake-connector-python[pandas]

$ poetry install --only main --extras snowflake --sync
Installing dependencies from lock file

Package operations: 27 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  [...]
  • Installing pandas (2.0.3)
  [...]
  • Installing snowflake-connector-python (3.3.1)

Installing the current project: group-extras (0.1.0)

This installs snowflake-connector-python[pandas,secure-local-storage], note that pandas is not getting removed

$ poetry install --with dev --extras snowflake --sync
Installing dependencies from lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing jeepney (0.8.0)
  • Installing more-itertools (10.1.0)
  • Installing zipp (3.17.0)
  • Installing importlib-metadata (6.8.0)
  • Installing jaraco-classes (3.3.0)
  • Installing secretstorage (3.3.3)
  • Installing keyring (24.2.0)

Installing the current project: group-extras (0.1.0)

This installs snowflake-connector-python[pandas] again, removing all dependencies from the secure-local-storage. At this point, I'd say that this works in a sane manner.

$ poetry install --only main --extras snowflake --sync
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 7 removals

  • Removing importlib-metadata (6.8.0)
  • Removing jaraco-classes (3.3.0)
  • Removing jeepney (0.8.0)
  • Removing keyring (24.2.0)
  • Removing more-itertools (10.1.0)
  • Removing secretstorage (3.3.3)
  • Removing zipp (3.17.0)

Installing the current project: group-extras (0.1.0)

If it were a bug or unintended behavior, I'd have expected that one dependency specification overwrites the other one. But since the listed extras from the dependency's definition in different groups are combined correctly, it feels like something that is safe to use. It's quite useful too, so shouldn't it be documented?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the extras section of the pyproject documentation and the dependency-groups documentation linked in the issue. Reproduce the shown poetry lock and poetry install commands with the sample pyproject configuration, then determine whether the behavior is supported or unintended. Done means documenting the supported behavior or clarifying the existing restriction after maintainer guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.