python-poetry / python-poetry/poetry

Add a dependency as editable and non-editable

Open
#8,219 5 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/feature status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

This is about installing a dependency in --editable mode for development while using the same dependency in the CI/CD pipeline from the package repository.

I think this discussion was done several times before. However, I still do not see a good solution, unless I missed something badly. Even the recent change with a more relaxed install process (https://github.com/python-poetry/poetry-core/pull/520) does not do the trick.

To give some context: I am developing a data service in a data engineering project.
My setup is to develop a service (proj-main) and some support libraries (proj-lib) with core data models and protobuf definitions. The development is carried out in two different workspaces. The file layout looks like this:

/workspaces/
  |- proj-lib/
  |- proj-main/

The split is due to the different scope and different artifact types: proj-lib is for downstream data users and shipped as a wheel. proj-main is to create a Docker image for deployment in k8s. proj-mainalso depends on proj-lib.

A pretty straight forward setup, I would say.

Having proj-lib added in editable mode (also in the git repository) leads to a broken pipeline, as proj-lib is not available in the next-door workspace in the build environment -- as we all know.

Adding proj-lib twice to pyproject.toml, one time as main dependency and the other time as dev dependence in --editable mode also does not do the trick, and -- absolutely correct -- leads to an error when configuring the setup due to version conflicts:

poetry add proj-lib
poetry add --group dev ../proj-lib --editable 

leads to

$ poetry add --group prod proj-lib
Using version ^0.7+g31a42f3 for proj-lib

Updating dependencies
Resolving dependencies... (0.0s)

Because proj-main depends on both proj-lib (0.0.0) @ file:///workspaces/proj-lib and proj-lib (^0.7+g31a42f3), version solving failed.

The relaxed install model for path dependency -- which I understood was meant to support this project layout -- does not help, as I need to add the dependency with the same name twice with different versions, which is obviously an inconsistent project description.

So my question: How to set this up in Poetry?

  • From some of the discussions that I read, I got the impression that there is an opinion that this setup is invalid/uncommon, anyhow. -> What would be a better setup taking the scope consideration into account?
  • Do I need to modify my pyproject.toml each time I check-in my proj-main?
  • Is there a better way to realize this setup, e.g., with markers?

In case that I am not badly wrong in my assumptions that this is currently an impossible setup: Wouldn't that be a feature for poetry, e.g., to have a marker that I can set when calling poetry install to select the editable proj-lib dependency?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dependency declarations and editable path dependency examples in pyproject.toml, then trace the poetry add and poetry install entry points involved in dependency resolution. Compare the reported proj-lib production and development setups and review the referenced poetry-core pull request 520. Done would require a decided, tested way to select editable or repository dependencies without conflicting declarations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.