Skaffold port forwarding gets into broken state after modifying artifact image name in Skaffold Config

1. Clone the [cloud-code-samples](https://github.com/GoogleCloudPlatform/cloud-code-samples) and go to the `golang/go-hello-world` dir.
2. Start a `skaffold dev` session using a command like `/Users/michihara/bin/skaffold dev --filename skaffold.yaml --label ide=Idea --label ideVersion=2021.1.3.0.0 --label ijPluginVersion=21.7.2-SNAPSHOT --rpc-port=50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true --enable-rpc=true`
3. The go-hello-world-external service should be port forwarded. A message like `Port forwarding service/go-hello-world-external in namespace default, remote port 80 -> http://127.0.0.1:4503` should appear in the logs. Clicking on http://127.0.0.1:4503 should lead to a sample hello world page.
4. Modify the image name in `skaffold.yaml`.
5. Skaffold should rebuild. http://127.0.0.1:4503 should no longer work.
6. Undo the image name change in `skaffold.yaml`.
7. Skaffold should rebuild. http://127.0.0.1:4503 still doesn't work although I expected it to.
8. Skaffold is now in a state where port forwarding no longer seems to work no matter what change is made.

https://user-images.githubusercontent.com/79073/130265519-45d661d5-317a-4945-8ddb-bfdda411e348.mov