With buildpack & auto sync a full re-build happens unpredictably

Tested using skaffold `1.17.2`

1) Clone the Cloud Code Go hello-world k8s sample: https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/golang/go-hello-world
2) Configure your build to use buildpacks:
```
build:
tagPolicy:
sha256: {}
artifacts:
- image: go-hello-world
context: .
buildpack:
builder: gcr.io/buildpacks/builder:v1
```
3) Run skaffold dev
4) Make a change in `main.go`

**Expected**: Skaffolds auto-sync for buildpacks would automtically sync the change without performing a full container rebuild.

**Actual**: The full build happens.

**Note**: This only seems to happen unpredictably; in my tests, some changes sync, others trigger a full rebuild (the changes are identical each time - e.g. a simple `fmt.Println` string modification in this case).

[Example output](https://paste.googleplex.com/6448961318551552) (the first couple changes did a full rebuild, the last change synced).