Not caching in multi-stage build

I have a Dockerfile utlising cache advantage of multi-stage build, and if I build it using docker cli it works perfectly. But if use docker-py to build the image using the same dockerfile, it re-installs the requirements using pip every time, even if there is no change in code directory.

Dockerfile

## Multi Stage build to take advantage of the Docker cache layers to speed up the builds

## Stage I

# Start from the official Python base image.
FROM python:3.8.10

# Set the current working directory to /code.
# This is where we'll put the requirements.txt file and the app directory.
WORKDIR /code

COPY install_functions.txt .
COPY copy_requirements.sh .
COPY functions functions
RUN ./copy_requirements.sh

## Stage II

FROM python:3.8.10

## Update and Install Packages
RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6 -y

## Update pip
ENV PIP_ROOT_USER_ACTION=ignore
ENV PIP_NO_CACHE_DIR=1
RUN pip install --upgrade pip

# Set the current working directory to /code.
# This is where we'll put the requirements.txt file and the app directory.
WORKDIR /code

# Install the package dependencies in the requirements file.
COPY --from=0 /code .
COPY install_functions.txt .
COPY install_requirements.sh .
RUN ./install_requirements.sh

# Copy the remaining functions to the /code directory.
COPY functions functions

Docker-cli command

docker build -t $tagname .

Output

Step 1/18 : FROM python:3.8.10
 ---> a369814a9797
Step 2/18 : WORKDIR /code
 ---> Using cache
 ---> 6050321ebdb3
Step 3/18 : COPY install_functions.txt .
 ---> Using cache
 ---> 9ec3c9862a9a
Step 4/18 : COPY copy_requirements.sh .
 ---> Using cache
 ---> 82b6e6bc1665
Step 5/18 : COPY functions functions
 ---> Using cache
 ---> 8eeae60ff70f
Step 6/18 : RUN ./copy_requirements.sh
 ---> Using cache
 ---> bd26459355ba
Step 7/18 : FROM python:3.8.10
 ---> a369814a9797
Step 8/18 : RUN apt-get update
 ---> Using cache
 ---> 3cf92e617eb0
Step 9/18 : RUN apt-get install ffmpeg libsm6 libxext6 -y
 ---> Using cache
 ---> 10cb11108f34
Step 10/18 : ENV PIP_ROOT_USER_ACTION=ignore
 ---> Using cache
 ---> 22aab820db77
Step 11/18 : ENV PIP_NO_CACHE_DIR=1
 ---> Using cache
 ---> 9b57f4673e40
Step 12/18 : RUN pip install --upgrade pip
 ---> Using cache
 ---> 874b82f5fd5f
Step 13/18 : WORKDIR /code
 ---> Using cache
 ---> 6256061d5bae
Step 14/18 : COPY --from=0 /code .
 ---> Using cache
 ---> 4f91640f93f8
Step 15/18 : COPY install_functions.txt .
 ---> Using cache
 ---> fea29b2f56da
Step 16/18 : COPY install_requirements.sh .
 ---> Using cache
 ---> 11db1813a9b2
Step 17/18 : RUN ./install_requirements.sh
 ---> Using cache
 ---> 7786c8b7aaf3
Step 18/18 : COPY functions functions
 ---> Using cache
 ---> 80a1da876432
Successfully built 80a1da876432

Python script

docker_client = docker.APIClient(base_url="unix://var/run/docker.sock")
docker_client.build(path=path, tag=tag, decode=True)

Output

Step 1/18 : FROM python:3.8.10

 ---> a369814a9797
Step 2/18 : WORKDIR /code

 ---> Using cache
 ---> 6050321ebdb3
Step 3/18 : COPY install_functions.txt .

 ---> 6ae19224e0e3
Step 4/18 : COPY copy_requirements.sh .

 ---> d35632789017
Step 5/18 : COPY functions functions

 ---> 5a4118f5bd03
Step 6/18 : RUN ./copy_requirements.sh

 ---> Running in 2ae618c78cc1
 Requirements corresponding to needed functions copied successfully 
 ---> fe49af04deb0
Step 7/18 : FROM python:3.8.10

 ---> a369814a9797
Step 8/18 : RUN apt-get update

 ---> Using cache
 ---> 3cf92e617eb0
Step 9/18 : RUN apt-get install ffmpeg libsm6 libxext6 -y

 ---> Using cache
 ---> 10cb11108f34
Step 10/18 : ENV PIP_ROOT_USER_ACTION=ignore

 ---> Using cache
 ---> 22aab820db77
Step 11/18 : ENV PIP_NO_CACHE_DIR=1

 ---> Using cache
 ---> 9b57f4673e40
Step 12/18 : RUN pip install --upgrade pip

 ---> Using cache
 ---> 874b82f5fd5f
Step 13/18 : WORKDIR /code

 ---> Using cache
 ---> 6256061d5bae
Step 14/18 : COPY --from=0 /code .

 ---> Using cache
 ---> 4f91640f93f8
Step 15/18 : COPY install_functions.txt .

 ---> f12aa2b9d99d
Step 16/18 : COPY install_requirements.sh .

 ---> afc723add0be
Step 17/18 : RUN ./install_requirements.sh

 ---> Running in d747e4fc7a10
Collecting numpy==1.23.2
  Downloading numpy-1.23.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 240.7 MB/s eta 0:00:00

Collecting pandas==1.4.4
  Downloading pandas-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 210.0 MB/s eta 0:00:00

Collecting python-dateutil==2.8.2
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
...... and so on

Any help would be appreciated.

Start with the docker.APIClient.build call and compare its request options with the docker build CLI invocation. Use the supplied Dockerfile and build output to trace why the COPY and RUN layers differ in cache behavior; done means repeated API builds reuse the same cached layers and do not reinstall the requirements.