Bug: Sam Build very slow and ignoring cache, recopying unchanged files and dependencies

### Description:

For context project is approx 40 lambdas, all python 3.8, using dependency layer for requirements.

Running `sam build` has become incredibly slow (10-15 min on my Macbook Pro) as project has grown and is really slowing down development process. Running `sam build --parallel --debug --cached` shows that most of the time seems to be taken during the `Copying source file` steps.

My lambda code is only about 1.5MB, the dependency layer is near 256MB. It appears that the Copy Source step is recopying the `.aws-sam/deps/{path to dependency cache file}` to the `.aws-sam/build/DependencyLayer/{path}` even when the dependency layer is unchanged.

This happens even if i just make a change to the template and no code changes, i still have to wait for this entire 10+ min build.

### Steps to reproduce:

Here is the relevant bit of my template for the dependency layer

```
PipDependencyLayer:
Type: AWS::Serverless::LayerVersion
Properties:
ContentUri: dependencies
CompatibleRuntimes:
- python3.8
Metadata:
BuildMethod: python3.8
```

I have a `dependencies` folder in my working directory root (same level as my project root)

```
dependencies
- python
- requirements.txt
project_root
- lambda_handler_1.py
- ... other project files
```

Requirements.txt content is below

```
-i https://pypi.org/simple
arabic-reshaper==2.1.4
asn1crypto==1.5.1
attrs==22.1.0; python_version >= '3.5'
aws-encryption-sdk==3.1.1
backports.zoneinfo==0.2.1; python_version < '3.9'
boto3==1.26.18; python_version >= '3.7'
botocore==1.29.18; python_version >= '3.7'
certifi==2022.9.24; python_version >= '3.6'
cffi==1.15.1
charset-normalizer==2.1.1; python_version >= '3.6'
click==8.1.3; python_version >= '3.7'
cryptography==38.0.4; python_version >= '3.6'
cssselect2==0.7.0; python_version >= '3.7'
future==0.18.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
html5lib==1.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
idna==3.4; python_version >= '3.5'
jmespath==1.0.1; python_version >= '3.7'
lxml==4.9.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
numpy==1.23.5
oscrypto==1.3.0
pandas==1.5.2
pillow==9.3.0
pycparser==2.21
pyhanko==0.15.1
pyhanko-certvalidator==0.19.6
pyjwt==2.6.0
pypdf3==1.0.6
python-bidi==0.4.2
python-dateutil==2.8.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-http-client==3.3.7; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-liquid==1.4.7
pytz==2022.6
pytz-deprecation-shim==0.1.0.post0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
pyyaml==6.0; python_version >= '3.6'
qrcode==7.3.1; python_version >= '3.6'
ratelimit==2.2.1
reportlab==3.6.12; python_version >= '3.7' and python_version < '4'
requests==2.28.1
s3transfer==0.6.0; python_version >= '3.7'
sendgrid==6.9.7
simplejson==3.18.0
six==1.16.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
starkbank-ecdsa==2.2.0
stripe==5.0.0
svglib==1.4.1; python_version >= '3.7'
tinycss2==1.2.1; python_version >= '3.7'
tqdm==4.64.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
typing-extensions==4.4.0; python_version >= '3.7'
tzdata==2022.6; python_version >= '3.6'
tzlocal==4.2; python_version >= '3.6'
uritools==4.0.0; python_version ~= '3.7'
urllib3==1.26.13; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
webencodings==0.5.1
wrapt==1.14.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
xhtml2pdf==0.2.8
```

### Observed result:

Described in the description, the build is very slow

### Expected result:

- I would expect that unchanged dependency layer would not be recopied into the `.aws-sam/build` directory even if lambda changes are made as long as no dependencies are updated
- I would expect that if no code changes are made and only template updates completed that there would be no build/copy source step necessary

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

output `sam --info`

```
{
"version": "1.76.0",
"system": {
"python": "3.8.16",
"os": "macOS-12.5.1-x86_64-i386-64bit"
},
"additional_dependencies": {
"docker_engine": "Not available",
"aws_cdk": "Not available",
"terraform": "Not available"
}
}
```