Detecting when a resource is thrashing

💔 **Describe the problem/challenge you have**
I noticed that sometimes my app deploys successfully, yet every time kapp-controller reconciles my app, it is changing something. I didn't notice this for a while, and even though you can figure out that this is happening from the information available in the `.status` field of my app, it wasn't immediately obvious for a beginner kapp user like me to deduce.

I've seen this happen in a couple scenarios, each where the initial deploy succeeds and led me to believe things were cool:
1. Webhook Configuration rules

When Knative MutatingWebhookConfiguration resources are applied to the cluster, like this [one](https://github.com/knative/serving/blob/main/config/core/webhooks/domainmapping-defaulting.yaml) for example, the `rules:` section will get replaced by Knative's webhook deployment (see this [line](https://github.com/knative/pkg/blob/main/webhook/resourcesemantics/validation/reconcile_config.go#L152)), to look something like this [gist](https://gist.github.com/KauzClay/8e73beacefea76b714f97b350772b2ac). So every reconciliation cycle, kapp-controller applies the first webhook, which then gets expanded on cluster, which then kapp-controller sees as a diff, and repeat. This isn't immediately obvious unless you are watching the status of your app and notice that there is a consistent presence of
```
Changes
Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
(cluster) webhook.serving.knative.dev MutatingWebhookConfiguration - 3m update - reconcile ok -
Op: 0 create, 0 delete, 1 update, 0 noop
Wait to: 1 reconcile, 0 delete, 0 noop
```
in `.status.deploy.stdout`.

2. Jobs with `ttlSecondsAfterFinished`

This [feature](https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/) is on by default in k8s v1.21+, and basically means that the job will clean up after itself after a specified time.

In certain scenarios, it is possible for the job to be reapplied multiple times by kapp-controller if it has deleted itself. I've tried to show this with this [app](https://gist.github.com/KauzClay/a49cb3e10ecfec588bf890219b2bf984). If you deploy it, hopefully you will see that a [job](https://github.com/KauzClay/carvel-simple-app-on-kubernetes/blob/develop/config-step-2-template/job.yml) is constantly being redeployed.

In this case, I will admit that the presence of `kapp.k14s.io/disable-wait: ""` annotation on the job enables this. Without it, the deploy will fail because the job is gone. However, in my real life experience with this, I didn't notice that the combo of that annotation and `ttlSecondsAfterFinished: 0` was having this affect for a while. And again, it wasn't immediately obvious until I watched the status of the app (similar to the example output above) and saw that each reconcile, a new job was getting created.

🔨 🔧 **Describe the solution you'd like**
I'm not sure if this situation warrants a full error/failed app, since the source of the problem comes from the contents of the app. But I can imagine a status field of something like `.status.consecutiveIdenticalDeploys` could be helpful as like an indicator light to the developer that something unintended might be going on.

For example, right now I have a bash script that watched the value of `.status.deploy.stdout` for a length of time, and checks to make sure that the fields
```
Op: 0 create, 0 delete, 1 update, 0 noop
Wait to: 1 reconcile, 0 delete, 0 noop
```
are all zeros. But that requires some specific `sed`/`regex` usage.

What I'd like to be able to do is just run a single `kubectl get app myapp -ojsonpath='{.status.consecutiveIdenticalDeploys}'` and use that to tell me if things are changing.

**Anything else you would like to add:**
~I don't totally understand why `1.` happens, but maybe it is worth making a separate issue for kapp-controller handling just this case?~
**EDIT:** for the situation described in 1., I learned that it is unique to Knative. Knative's webhook deployment does the expanding of the config, not anything part of baseline k8s.

---
Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.