Template level lifecycle hooks unable to access workflow.status

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

In Argo v3.4.8 the template level lifecycle hooks fail to trigger when using `workflow.status`

The reason I am using a template level lifecycle hook instead of just an onExit handler, is that I want to able to destroy a `Resource` created by my workflow (such as a PVC) even if the workflow fails in the middle of execution.

See example use case

```yaml
metadata:
name: dynamic-pvc-6794c
spec:
templates:
- name: whalesay
inputs:
parameters:
- name: i
- name: pvc-name
outputs: {}
metadata: {}
container:
name: ''
image: docker/whalesay:latest
command:
- sh
- '-c'
args:
- >-
echo generating message in volume {{inputs.parameters.pvc-name}};
cowsay hello world | tee
/mnt/vol/hello_world-{{inputs.parameters.i}}.txt
resources: {}
volumeMounts:
- name: workdir
mountPath: /mnt/vol
volumes:
- name: workdir
persistentVolumeClaim:
claimName: '{{inputs.parameters.pvc-name}}'
- name: print-message
inputs:
parameters:
- name: i
- name: pvc-name
outputs: {}
metadata: {}
container:
name: ''
image: alpine:latest
command:
- sh
- '-c'
args:
- >-
echo getting message from volume; find /mnt/vol; cat
/mnt/vol/hello_world-{{inputs.parameters.i}}.txt
resources: {}
volumeMounts:
- name: workdir
mountPath: /mnt/vol
volumes:
- name: workdir
persistentVolumeClaim:
claimName: '{{inputs.parameters.pvc-name}}'
- name: generate-pvc
inputs:
parameters:
- name: pvc-size
outputs:
parameters:
- name: pvc-name
valueFrom:
jsonPath: '{.metadata.name}'
metadata: {}
resource:
action: create
manifest: |-
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
generateName: hera-scratch-pvc-
spec:
accessModes: ['ReadWriteOnce']
resources:
requests:
storage: '{{inputs.parameters.pvc-size}}'
storageClassName: standard-rwo

setOwnerReference: true
- name: delete-pvc
inputs:
parameters:
- name: pvc-name
outputs: {}
metadata: {}
resource:
action: delete
flags:
- pvc
- '{{inputs.parameters.pvc-name}}'
- '--grace-period=0'
- '--force'
- name: worker
inputs:
parameters:
- name: i
outputs: {}
metadata: {}
steps:
- - name: generate-volume
template: generate-pvc
arguments:
parameters:
- name: pvc-size
value: 1Gi
# This hook fails to trigger
hooks:
exit:
template: delete-pvc
arguments: {}
expression: >-
(workflow.status == "Error") || (workflow.status == "Failed")
||(workflow.status == "Succeeded")
- - name: whalesay
template: whalesay
arguments:
parameters:
- name: i
value: '{{inputs.parameters.i}}'
- name: pvc-name
value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
- - name: print
template: print-message
arguments:
parameters:
- name: i
value: '{{inputs.parameters.i}}'
- name: pvc-name
value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
- - name: delete-pvc
template: delete-pvc
arguments:
parameters:
- name: pvc-name
value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
- name: entrypoint
inputs: {}
outputs: {}
metadata: {}
steps:
- - name: parallel-worker-0
template: worker
arguments:
parameters:
- name: i
value: '0'
- name: parallel-worker-1
template: worker
arguments:
parameters:
- name: i
value: '1'
- name: parallel-worker-2
template: worker
arguments:
parameters:
- name: i
value: '2'
entrypoint: entrypoint
arguments: {}
strategy: OnPodCompletion
```

### Version

v3.3.8

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: lifecycle-hook-tmpl-level-
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: step-1
hooks:
running: # Name of hook does not matter
# Expr will not support `-` on variable name. Variable should wrap with `[]`
expression: steps["step-1"].status == "Running"
template: echo
success:
expression: steps["step-1"].status == "Succeeded"
template: echo

# None of the below hooks get triggered
workflow-running:
expression: workflow.status == "Running"
template: echo
workflow-done:
expression: workflow.status == "Succeeded"
template: echo
workflow-failed:
expression: workflow.status == "Failed"
template: echo
workflow-error:
expression: workflow.status == "Error"
template: echo
template: echo

- name: echo
container:
image: alpine:3.6
command: [sh, -c]
args: ["echo \"it was heads\""]
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2023-06-26T11:52:09.896Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="Updated phase -> Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="Steps node lifecycle-hook-tmpl-level-zm7zf initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="StepGroup node lifecycle-hook-tmpl-level-zm7zf-4058935595 initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.901Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-3586332675 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.923Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1 (lifecycle-hook-tmpl-level-zm7zf-echo-3586332675)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.937Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858033 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.924Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-3586332675 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-3586332675 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.926Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.926Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-156141424 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.948Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success (lifecycle-hook-tmpl-level-zm7zf-echo-156141424)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.950Z" level=info msg="Running hooks" hookName=workflow-running lifeCycleHook=workflow-running namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.951Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-4170442462 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running (lifecycle-hook-tmpl-level-zm7zf-echo-4170442462)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.976Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858139 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:24.977Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-3586332675/deletePod
time="2023-06-26T11:52:29.948Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="Task-result reconciliation" namespace=default numObjs=2 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-156141424 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-4170442462 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-4170442462 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-156141424 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Running hooks" hookName=workflow-running lifeCycleHook=workflow-running namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 successful" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-4058935595 phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-4058935595 finished: 2023-06-26 11:52:29.950516792 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="StepGroup node lifecycle-hook-tmpl-level-zm7zf-904993486 initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-512981488 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.962Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[1].step2 (lifecycle-hook-tmpl-level-zm7zf-echo-512981488)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-904993486 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.980Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858252 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:34.981Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-156141424/deletePod
time="2023-06-26T11:52:34.981Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-4170442462/deletePod
time="2023-06-26T11:52:39.964Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-512981488 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-512981488 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-1227685471 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success (lifecycle-hook-tmpl-level-zm7zf-echo-1227685471)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-904993486 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.999Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858357 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:45.000Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-512981488/deletePod
time="2023-06-26T11:52:49.985Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.986Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.986Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-1227685471 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-1227685471 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="Step group node lifecycle-hook-tmpl-level-zm7zf-904993486 successful" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-904993486 phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-904993486 finished: 2023-06-26 11:52:49.987975644 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-512981488 is [lifecycle-hook-tmpl-level-zm7zf-512981488]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Outbound nodes of lifecycle-hook-tmpl-level-zm7zf is [lifecycle-hook-tmpl-level-zm7zf-512981488]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf finished: 2023-06-26 11:52:49.988032529 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Checking daemoned children of lifecycle-hook-tmpl-level-zm7zf" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Marking workflow completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Checking daemoned children of " namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.993Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-1340600742-agent/deletePod
time="2023-06-26T11:52:50.001Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=181858458 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:50.001Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h59m59s due to TTL"
time="2023-06-26T11:52:55.026Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-1227685471/deletePod
time="2023-06-26T12:24:17.471Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h28m32s due to TTL"
time="2023-06-26T12:44:17.472Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h8m32s due to TTL"
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

None
```

Start with the workflow controller's lifecycle-hook reconciliation using the minimal YAML reproduction, then compare the controller logs for the successful hook and the workflow.status hooks. Reproduce the issue on the reported Argo versions and trace why workflow.status expressions do not trigger; done means the relevant template-level hooks run for the expected workflow phases without breaking existing hooks.

Written by the indexing model from the issue text.