Cannot install Monorepo deps without sourcecode for Dockerfile caching

Description

Hi all,

I'm having difficulty setting up a proper Dockerfile for a service in a monorepo that doesn't require the source code for the services and libraries included in the monorepo.

The monorepo has a shared venv. There is a root pyproject.toml and a few packages/libraries and services nested, each with their own pyproject.toml.

My root pyproject.toml looks like this:

# ./pyproject.toml

[tool.poetry.group.main.dependencies]
# all the shared deps

[tool.poetry.group.my-library.dependencies]
my-library = { path = "packages/my-library", develop = true }

[tool.poetry.group.my-service.dependencies]
my-service = { path = "services/my-service", develop = true }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

and my service will depend on a library as such:

# services/my-service/pyproject.toml
[tool.poetry.dependencies]
python = "3.11.4"
my-library = { path = "../../packages/my-library" }
# my-service's specific deps

I'm trying to optimize my Dockerbuild so that I don't have to rebuild it everytime I have some source changes. Here's what I'd like to do:

FROM python-base-image

RUN apt-get update && \
  apt-get install -y supervisor poppler-utils && \
  apt-get install --no-install-suggests --no-install-recommends --yes python3-venv pipx

ENV PATH="/root/.local/bin:${PATH}"
RUN pipx install poetry

# Copy poetry and pyproject files over so deps can be installed
COPY ./poetry.lock /home/app/poetry.lock
COPY ./poetry.toml /home/app/poetry.toml
COPY ./pyproject.toml /home/app/pyproject.toml

COPY ./packages/my-library/pyproject.toml /home/app/packages/my-library/pyproject.toml
COPY ./services/my-service/pyproject.toml /home/app/services/my-service/pyproject.toml


WORKDIR /home/app
RUN poetry lock --no-update
RUN poetry install --no-root --no-directory --only my-service,main

# Copy source code over
COPY ./packages /home/app/packages
COPY ./services/my-service /home/app/services/my-service 

My understanding is that --no-root --no-directory will allow me to install my service's deps, including my-library, without needing the source code. This way, I can utilize Docker layers so I don't have to rebuild when I have source code changes.

However, I keep getting this error, and I'm unsure of how to fix it - could I get some guidance?

#0 425.8   • Installing my-library (0.1.0 /home/app/packages/my-library)
#0 432.8 
#0 432.8   ChefBuildError
#0 432.8 
#0 432.8   Backend subprocess exited when trying to invoke build_editable
#0 432.8   
#0 432.8   Traceback (most recent call last):
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
#0 432.8       main()
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
#0 432.8       json_out["return_val"] = hook(**hook_input["kwargs"])
#0 432.8                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 294, in build_editable
#0 432.8       return hook(wheel_directory, config_settings, metadata_directory)
#0 432.8              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 82, in build_editable
#0 432.8       return WheelBuilder.make_in(
#0 432.8              ^^^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
#0 432.8       wb.build(target_dir=directory)
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
#0 432.8       self._add_pth(zip_file)
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 147, in _add_pth
#0 432.8       for include in self._module.includes:
#0 432.8                      ^^^^^^^^^^^^
#0 432.8     File "/root/.pyenv/versions/3.11.4/lib/python3.11/functools.py", line 1001, in __get__
#0 432.8       val = self.func(instance)
#0 432.8             ^^^^^^^^^^^^^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/builder.py", line 97, in _module
#0 432.8       return Module(
#0 432.8              ^^^^^^^
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/module.py", line 75, in __init__
#0 432.8       PackageInclude(
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 31, in __init__
#0 432.8       self.check_elements()
#0 432.8     File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 72, in check_elements
#0 432.8       raise ValueError(
#0 432.8   ValueError: /home/app/packages/my-library/my-library does not contain any element
#0 432.8   
#0 432.8 
#0 432.8   at ~/.local/pipx/venvs/poetry/lib/python3.7/site-packages/poetry/installation/chef.py:147 in _prepare
#0 432.8       143│ 
#0 432.8       144│                 error = ChefBuildError("\n\n".join(message_parts))
#0 432.8       145│ 
#0 432.8       146│             if error is not None:
#0 432.8     → 147│                 raise error from None
#0 432.8       148│ 
#0 432.8       149│             return path
#0 432.8       150│ 
#0 432.8       151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#0 432.8 
#0 432.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with my-library (0.1.0 /home/app/packages/my-library) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 --editable "/home/app/packages/my-library"'.

If I copy the sources over before running poetry install, I don't have the issue - but I can't utilize Docker layering.

Thank you guys very much!

PS - I'm also trying to use Docker's cache mount, but Poetry seems to ignore it per the Poetry Config. but that's for another day.

ENV POETRY_CACHE_DIR='/home/.cache/pypoetry'
# ...
RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install --no-root --no-directory --only my-service,main

Workarounds

COPY the source for my library and service prior to poetry install - but this is undesired and slow

Poetry Installation Method

pipx

Operating System

Debian

Poetry Version

Poetry 1.5.1

Poetry Configuration

In docker container:


# poetry config --list
cache-dir = "/root/ cache/pypoetry" # /home/ .cache/pypoetry
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir)/virtualenvs" # /home/.cache/pypoetry/virtualenvs/
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "[project_name)-py{python_version}"

### Python Sysconfig

_No response_

### Example pyproject.toml

_No response_

### Poetry Runtime Logs

```bash session
n/a

Start by reproducing the Dockerfile sequence with the root and nested pyproject.toml files, poetry.lock, and poetry.toml, focusing on poetry install --no-root --no-directory and the editable PEP 517 build error. Trace how path dependencies are handled during installation; done means dependencies install without copying package source first, while retaining Docker cache layers for later source changes.