Slow Rebuilds Using Gradle+Skaffold+Buildpacks

### Expected behavior

When using "skaffold build" image rebuilds with buildpacks (after an initial successful build) should be fast (<10s ideally).

### Actual behavior

Image rebuilds take 1m+. When switching to use the Jib builder, builds are consistently sub 10s.

### Information

I have a Spring Boot-based application with a Gradle-based build process. The Spring Boot Gradle plug-in includes support for creating buildpacks based images, and these are what we are currently deploying into production. When adopting skaffold, we defaulted to using the buildpacks builder as well for consistency, but have found it to be noticeably slower than other options (specifically Jib, or rebuilding without skaffold).

Comparisons below. These are based on rebuilding the project with a minor code change (eg. changing a string from "Hello World!" to "Hello World!!"):
- running `./gradlew bootBuildImage` (which runs the buildpacks builder to create a new image): 9s
- running `skaffold build` with Jib: 5s
- running `skaffold build` with buildpacks: 1m 20s

Further details:
- Skaffold version: 1.35.1
- Operating system: Ubuntu 20.04 running on WSL2 (Windows 10)
- Installed via: Homebrew
- Contents of skaffold.yaml:

```yaml
apiVersion: skaffold/v2beta23
kind: Config
metadata:
name: skaffold-boot

build:
local:
push: false
artifacts:
# - image: skaffold-boot
# context: .
# jib: {}
- image: skaffold-boot
buildpacks:
builder: paketobuildpacks/builder:base

deploy:
kubeContext: minikube
kustomize:
paths:
- k8s/local

portForward:
- resourceType: deployment
resourceName: skaffold-boot
namespace: default
port: 8080
localPort: 8080

```

### Steps to reproduce the behavior

Sample project: https://github.com/awesometown/skaffold-boot

Logs for the various run types are in the linked repository.

The skaffold.yaml in the project contains two builder configs, one using buildpacks, one for jib. You can uncomment the appropriate one and re-run `skaffold build` after making a trivial change to the source code.

You can also see the performance of creating a buldpacks build without skaffold by running `./gradlew bootBuildImage`