Notifications intermittently not being sent when notification queue is overloaded and/or notification services are slow to respond

**Describe the bug**
Notification engine fails to remove `on-deployed` key from `notified.notifications.argoproj.io` annotation before ArgoCD app state moves to **Synced** & **Healthy**

This happens when notification-engine picks up an app from the queue to process after the states already move to **Synced** and **Healthy**. It could happen when the queue is huge and/or notification services are slow to respond.

**Our system**
number of applications: >160
notification services:
- slack
- opslevel (webhook)
- internal deployment registry (webhook)

`trigger.on-deployed: |
- description: Application is synced and healthy.
send:
- app-deployed
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
`

**Reproducible steps**
1. add (1) a slack notification service and (2) a notification service that delays its response (ie. 30s)

` service.slack: |
token: $slack-token
icon: ":argo:"
signingSecret: $slack-signing-secret
service.webhook.test: |
url:
headers:
- name: X-Delay-Duration
value: 30s
`

2. annotate all apps with on-sync-running, on-sync-succeeded, on-deployed

k annotate application --all -n argocd notifications.argoproj.io/subscribe.on-sync-running.test=""
k annotate application --all -n argocd notifications.argoproj.io/subscribe.on-sync-succeeded.test=""
k annotate application --all -n argocd notifications.argoproj.io/subscribe.on-deployed.test=""
k annotate application --all -n argocd notifications.argoproj.io/subscribe.on-deployed.slack="#my-channel"

Note: this will make notification-engine stuck in waiting state for the response from the slow-notification-service during sync-running and sync-succeeded while argocd completes sync & rollout of the test app, whose states have already transition to Synced and Healthy.

PS. this issue can occur easily with
- more apps (with frequent state changes)
- more notification services (with slow response)
- smaller sync window (from OutOfSync to Synced & Healthy)

3. update target revision of a test app

**Version**
notifications-engine: [v0.3.1](https://github.com/argoproj/argo-cd/blob/v2.4.16/go.mod#L13)
argocd:
{
"Version": "v2.4.16+7b5899b",
"BuildDate": "2022-11-01T21:17:46Z",
"GitCommit": "7b5899be33d16af7c57523d85ebacaa6f345cb95",
"GitTreeState": "clean",
"GoVersion": "go1.18.8",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
"HelmVersion": "v3.8.1+g5cb9af4",
"KubectlVersion": "v0.23.1",
"JsonnetVersion": "v0.18.0"
}

Start by tracing notification-engine queue processing and the handling of the `notified.notifications.argoproj.io` annotation while reproducing the issue with a delayed webhook and multiple applications. Done means the `on-deployed` key is removed and the notification is sent when the application reaches Synced and Healthy, even when earlier notifications are slow.

Written by the indexing model from the issue text.