python-poetry / python-poetry/poetry
Add a dependency as editable and non-editable
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.3k
- Forks
- 2.5k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 30
Description
This is about installing a dependency in --editable mode for development while using the same dependency in the CI/CD pipeline from the package repository.
I think this discussion was done several times before. However, I still do not see a good solution, unless I missed something badly. Even the recent change with a more relaxed install process (https://github.com/python-poetry/poetry-core/pull/520) does not do the trick.
To give some context: I am developing a data service in a data engineering project.
My setup is to develop a service (proj-main) and some support libraries (proj-lib) with core data models and protobuf definitions. The development is carried out in two different workspaces. The file layout looks like this:
/workspaces/
|- proj-lib/
|- proj-main/
The split is due to the different scope and different artifact types: proj-lib is for downstream data users and shipped as a wheel. proj-main is to create a Docker image for deployment in k8s. proj-mainalso depends on proj-lib.
A pretty straight forward setup, I would say.
Having proj-lib added in editable mode (also in the git repository) leads to a broken pipeline, as proj-lib is not available in the next-door workspace in the build environment -- as we all know.
Adding proj-lib twice to pyproject.toml, one time as main dependency and the other time as dev dependence in --editable mode also does not do the trick, and -- absolutely correct -- leads to an error when configuring the setup due to version conflicts:
poetry add proj-lib
poetry add --group dev ../proj-lib --editable
leads to
$ poetry add --group prod proj-lib
Using version ^0.7+g31a42f3 for proj-lib
Updating dependencies
Resolving dependencies... (0.0s)
Because proj-main depends on both proj-lib (0.0.0) @ file:///workspaces/proj-lib and proj-lib (^0.7+g31a42f3), version solving failed.
The relaxed install model for path dependency -- which I understood was meant to support this project layout -- does not help, as I need to add the dependency with the same name twice with different versions, which is obviously an inconsistent project description.
So my question: How to set this up in Poetry?
- From some of the discussions that I read, I got the impression that there is an opinion that this setup is invalid/uncommon, anyhow. -> What would be a better setup taking the scope consideration into account?
- Do I need to modify my
pyproject.tomleach time I check-in myproj-main? - Is there a better way to realize this setup, e.g., with markers?
In case that I am not badly wrong in my assumptions that this is currently an impossible setup: Wouldn't that be a feature for poetry, e.g., to have a marker that I can set when calling poetry install to select the editable proj-lib dependency?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dependency declarations and editable path dependency examples in pyproject.toml, then trace the poetry add and poetry install entry points involved in dependency resolution. Compare the reported proj-lib production and development setups and review the referenced poetry-core pull request 520. Done would require a decided, tested way to select editable or repository dependencies without conflicting declarations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100