Trigger workflow with Azure Eventhub Sensor: `expected < 2 pods, got 2 - this is a bug`

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [X] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

I tried to implement execute a workflow from workflow template with event trigger. I published an event that contained the workflow template name. In Sensor, I firstly decode event and then used Argo submit command to submit this workflow template in customized image that enable Argo CLI. The workflow started successfully. However, it caused error: `
expected < 2 pods, got 2 - this is a bug`.
I don't know what happened. I seemed that this step run successfully from log.

Why I used Argo submit in Container? because the TemplateRef property don't support variable coming from event.

### Version

v3.2.9

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: eventhub-sensor
spec:
template:
serviceAccountName: default
dependencies:
- name: dependent-eventsource
eventSourceName: eventhub-eventsource
eventName: eventName
triggers:
- template:
name: trigger-eventhub
k8s:
operation: create
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: eventhub-trigger-
spec:
arguments:
parameters:
- name: message
entrypoint: eventhub-trigger
templates:
- name: eventhub-trigger
inputs:
parameters:
- name: message
steps:
- - name: extract-event-info
template: event-info-extractor
arguments:
parameters:
- name: message
value: '{{inputs.parameters.message}}'
- - name: trigger-workflow
template: workflow-trigger
arguments:
parameters:
- name: triggerd-workflow
value: '{{steps.extract-event-info.outputs.parameters.triggerd-workflow}}'
- name: event-info-extractor
inputs:
parameters:
- name: message
script:
image: python:alpine3.6
imagepullpolicy: Always
command: [python]
source: |
import json
import base64
import os
message = base64.b64decode("{{inputs.parameters.message}}").decode("utf-8")
message_json = json.loads(message)
print("message", message_json)
os.makedirs('/tmp', exist_ok=True)
with open("/tmp/triggerd_workflow.txt", "w+") as f:
f.write(message_json["triggered_workflow"])
outputs:
parameters:
- name: triggerd-workflow
valueFrom:
path: /tmp/triggerd_workflow.txt
- name: scenario-id
valueFrom:
path: /tmp/scenario_id.txt
- name: workflow-trigger
inputs:
parameters:
- name: triggerd-workflow
container:
image: xxx // image that enables Argo-Cli
imagepullpolicy: Always
command: [sh, -c]
args: ["argo submit --from workflowtemplate/{{inputs.parameters.triggerd-workflow}}"]
parameters:
- src:
dependencyName: dependent-eventsource
dataKey: body
dest: spec.arguments.parameters.0.value
```

### Logs from the workflow controller

```
time="2022-10-16T09:30:42.824Z" level=info msg="Processing workflow" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.911Z" level=info msg="Updated phase -> Running" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.911Z" level=info msg="Steps node workflow-template-steps-p5l2l initialized Running"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.912Z" level=info msg="StepGroup node workflow-template-steps-p5l2l-1338817549 initialized
Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.912Z" level=info msg="Pod node workflow-template-steps-p5l2l-983838386 initialized Pending"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.954Z" level=info msg="Created pod: workflow-template-steps-p5l2l[0].hello1
(workflow-template-steps-p5l2l-983838386)" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.954Z" level=info msg="Workflow step group node workflow-template-steps-p5l2l-1338817549 not
yet completed" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.954Z" level=info msg="TaskSet Reconciliation" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.954Z" level=info msg=reconcileAgentPod namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:42.975Z" level=info msg="Workflow update successful" namespace=argo-main phase=Running
resourceVersion=18124162 workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.955Z" level=info msg="Processing workflow" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.955Z" level=info msg="Updating node workflow-template-steps-p5l2l-983838386 exit code 0"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.955Z" level=info msg="Updating node workflow-template-steps-p5l2l-983838386 status Pending
-> Succeeded" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.956Z" level=info msg="Step group node workflow-template-steps-p5l2l-1338817549 successful"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.956Z" level=info msg="node workflow-template-steps-p5l2l-1338817549 phase Running ->
Succeeded" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.956Z" level=info msg="node workflow-template-steps-p5l2l-1338817549 finished: 2022-10-16
09:30:52.956634434 +0000 UTC" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.956Z" level=info msg="StepGroup node workflow-template-steps-p5l2l-1271854168 initialized
Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.956Z" level=info msg="SG Outbound nodes of workflow-template-steps-p5l2l-983838386 are
[workflow-template-steps-p5l2l-983838386]" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.957Z" level=info msg="Steps node workflow-template-steps-p5l2l-1224969675 initialized
Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.957Z" level=info msg="StepGroup node workflow-template-steps-p5l2l-737134567 initialized
Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:52.957Z" level=info msg="Pod node workflow-template-steps-p5l2l-2600658835 initialized Pending"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.016Z" level=info msg="Created pod: workflow-template-steps-p5l2l[1].hello2a[0].inner-hello1
(workflow-template-steps-p5l2l-2600658835)" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.016Z" level=info msg="Workflow step group node workflow-template-steps-p5l2l-737134567 not
yet completed" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.016Z" level=info msg="Pod node workflow-template-steps-p5l2l-1241747294 initialized Pending"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.073Z" level=info msg="Created pod: workflow-template-steps-p5l2l[1].hello2b
(workflow-template-steps-p5l2l-1241747294)" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.073Z" level=info msg="Workflow step group node workflow-template-steps-p5l2l-1271854168 not
yet completed" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.073Z" level=info msg="TaskSet Reconciliation" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.073Z" level=info msg=reconcileAgentPod namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.109Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on
workflows.argoproj.io \"workflow-template-steps-p5l2l\": the object has been modified; please apply your changes to
the latest version and try again Conflict" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.109Z" level=info msg="Re-applying updates on latest version and retrying update"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.157Z" level=info msg="Update retry attempt 1 successful" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.157Z" level=info msg="Workflow update successful" namespace=argo-main phase=Running
resourceVersion=18124281 workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:30:53.164Z" level=info msg="cleaning up pod" action=labelPodCompleted
key=argo-main/workflow-template-steps-p5l2l-whalesay-template-983838386/labelPodCompleted
time="2022-10-16T09:30:53.164Z" level=info msg="cleaning up pod" action=labelPodCompleted
key=argo-main/workflow-template-steps-p5l2l-983838386/labelPodCompleted
time="2022-10-16T09:31:03.018Z" level=info msg="Processing workflow" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.019Z" level=info msg="Updating node workflow-template-steps-p5l2l-2600658835 status Pending
-> Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.019Z" level=info msg="Updating node workflow-template-steps-p5l2l-2600658835 exit code 0"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.019Z" level=info msg="Updating node workflow-template-steps-p5l2l-2600658835 status Running
-> Succeeded" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.019Z" level=info msg="Updating node workflow-template-steps-p5l2l-1241747294 status Pending
-> Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.020Z" level=info msg="SG Outbound nodes of workflow-template-steps-p5l2l-983838386 are
[workflow-template-steps-p5l2l-983838386]" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.021Z" level=info msg="Step group node workflow-template-steps-p5l2l-737134567 successful"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.021Z" level=info msg="node workflow-template-steps-p5l2l-737134567 phase Running ->
Succeeded" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.021Z" level=info msg="node workflow-template-steps-p5l2l-737134567 finished: 2022-10-16
09:31:03.021253364 +0000 UTC" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.021Z" level=info msg="StepGroup node workflow-template-steps-p5l2l-804392138 initialized
Running" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.021Z" level=info msg="SG Outbound nodes of workflow-template-steps-p5l2l-2600658835 are
[workflow-template-steps-p5l2l-2600658835]" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.022Z" level=info msg="Pod node workflow-template-steps-p5l2l-3622003880 initialized Pending"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.113Z" level=info msg="Created pod: workflow-template-steps-p5l2l[1].hello2a[1].inner-hello2a
(workflow-template-steps-p5l2l-3622003880)" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.114Z" level=info msg="Pod node workflow-template-steps-p5l2l-3672336737 initialized Pending"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="Created pod: workflow-template-steps-p5l2l[1].hello2a[1].inner-hello2b
(workflow-template-steps-p5l2l-3672336737)" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="Workflow step group node workflow-template-steps-p5l2l-804392138 not
yet completed" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=error msg="Mark error node" error="expected < 2 pods, got 2 - this is a bug"
namespace=argo-main nodeName="workflow-template-steps-p5l2l[1].hello2b" workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1241747294 phase Running -> Error"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1241747294 message: expected < 2
pods, got 2 - this is a bug" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1241747294 finished: 2022-10-16
09:31:03.157672565 +0000 UTC" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=error msg="Mark error node" error="step group deemed errored due to child
workflow-template-steps-p5l2l[1].hello2b error: expected < 2 pods, got 2 - this is a bug" namespace=argo-main
nodeName="workflow-template-steps-p5l2l[1]" workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1271854168 phase Running -> Error"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1271854168 message: step group
deemed errored due to child workflow-template-steps-p5l2l[1].hello2b error: expected < 2 pods, got 2 - this is a bug"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l-1271854168 finished: 2022-10-16
09:31:03.157773414 +0000 UTC" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="step group workflow-template-steps-p5l2l-1271854168 was unsuccessful:
step group deemed errored due to child workflow-template-steps-p5l2l[1].hello2b error: expected < 2 pods, got 2 - this
is a bug" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="Outbound nodes of workflow-template-steps-p5l2l-1224969675 is []"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="Outbound nodes of workflow-template-steps-p5l2l-1241747294 is
[workflow-template-steps-p5l2l-1241747294]" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="Outbound nodes of workflow-template-steps-p5l2l is
[workflow-template-steps-p5l2l-1241747294]" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l phase Running -> Failed"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l message: step group deemed errored
due to child workflow-template-steps-p5l2l[1].hello2b error: expected < 2 pods, got 2 - this is a bug"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.157Z" level=info msg="node workflow-template-steps-p5l2l finished: 2022-10-16
09:31:03.157985506 +0000 UTC" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="Checking daemoned children of workflow-template-steps-p5l2l"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="TaskSet Reconciliation" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg=reconcileAgentPod namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="Updated phase Running -> Failed" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="Updated message -> step group deemed errored due to child
workflow-template-steps-p5l2l[1].hello2b error: expected < 2 pods, got 2 - this is a bug" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="Marking workflow completed" namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.158Z" level=info msg="Checking daemoned children of " namespace=argo-main
workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.190Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on
workflows.argoproj.io \"workflow-template-steps-p5l2l\": the object has been modified; please apply your changes to
the latest version and try again Conflict" namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.190Z" level=info msg="Re-applying updates on latest version and retrying update"
namespace=argo-main workflow=workflow-template-steps-p5l2l
time="2022-10-16T09:31:03.200Z" level=info msg="Failed to re-apply update: must never update completed workflows"
namespace=argo-main workflow=workflow-template-steps-p5l2l
```

### Logs from in your workflow's wait container

```
time="2022-10-16T09:30:55.716Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.2.9
time="2022-10-16T09:30:55.722Z" level=info msg="Creating a emissary executor"
time="2022-10-16T09:30:55.723Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-2600658835 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello1\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello1\"],\"resources\":{}}}" version="&Version{Version:v3.2.9,BuildDate:2022-03-02T21:41:01Z,GitCommit:ce91d7b1d0115d5c73f6472dca03ddf5cc2c98f4,GitTag:v3.2.9,GitTreeState:clean,GoVersion:go1.16.14,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:55.723Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:30:56.723Z" level=info msg="Main container completed"
time="2022-10-16T09:30:56.723Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:30:56.723Z" level=info msg="No output parameters"
time="2022-10-16T09:30:56.723Z" level=info msg="No output artifacts"
time="2022-10-16T09:30:56.723Z" level=info msg="Killing sidecars []"
time="2022-10-16T09:30:56.723Z" level=info msg="Alloc=4857 TotalAlloc=9485 Sys=74065 NumGC=3 Goroutines=6"
time="2022-10-16T09:31:05.959Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.2.9
time="2022-10-16T09:31:05.969Z" level=info msg="Creating a emissary executor"
time="2022-10-16T09:31:05.969Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-3672336737 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello2b\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello2b\"],\"resources\":{}}}" version="&Version{Version:v3.2.9,BuildDate:2022-03-02T21:41:01Z,GitCommit:ce91d7b1d0115d5c73f6472dca03ddf5cc2c98f4,GitTag:v3.2.9,GitTreeState:clean,GoVersion:go1.16.14,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:31:05.969Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:07.970Z" level=info msg="Main container completed"
time="2022-10-16T09:31:07.970Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:07.970Z" level=info msg="No output parameters"
time="2022-10-16T09:31:07.970Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:07.970Z" level=info msg="Killing sidecars []"
time="2022-10-16T09:31:07.970Z" level=info msg="Alloc=4295 TotalAlloc=9435 Sys=73809 NumGC=4 Goroutines=6"
time="2022-10-16T09:30:55.738Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-10-16T09:30:55.738Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-whalesay-template-1241747294 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello2b\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"hello2b\"],\"resources\":{}}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:55.738Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:03.741Z" level=info msg="Main container completed" error=""
time="2022-10-16T09:31:03.741Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:03.741Z" level=info msg="No output parameters"
time="2022-10-16T09:31:03.741Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:03.741Z" level=info msg="Deadline monitor stopped"
time="2022-10-16T09:31:03.741Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-10-16T09:31:03.741Z" level=info msg="Alloc=6468 TotalAlloc=12087 Sys=23762 NumGC=4 Goroutines=6"
time="2022-10-16T09:31:05.973Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-10-16T09:31:05.973Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-whalesay-template-3672336737 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello2b\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello2b\"],\"resources\":{}}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:31:05.974Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:07.974Z" level=info msg="Main container completed" error=""
time="2022-10-16T09:31:07.974Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:07.974Z" level=info msg="No output parameters"
time="2022-10-16T09:31:07.974Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:07.974Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-10-16T09:31:07.974Z" level=info msg="Deadline monitor stopped"
time="2022-10-16T09:31:07.974Z" level=info msg="Alloc=6864 TotalAlloc=12029 Sys=19410 NumGC=3 Goroutines=6"
time="2022-10-16T09:30:44.494Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-10-16T09:30:44.498Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-10-16T09:30:44.498Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-whalesay-template-983838386 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello1\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"hello1\"],\"resources\":{}}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:44.499Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:30:46.499Z" level=info msg="Main container completed" error=""
time="2022-10-16T09:30:46.499Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:30:46.499Z" level=info msg="No output parameters"
time="2022-10-16T09:30:46.499Z" level=info msg="No output artifacts"
time="2022-10-16T09:30:46.499Z" level=info msg="Alloc=6460 TotalAlloc=12337 Sys=24530 NumGC=4 Goroutines=6"
time="2022-10-16T09:30:55.719Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.2.9
time="2022-10-16T09:30:55.725Z" level=info msg="Creating a emissary executor"
time="2022-10-16T09:30:55.725Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-1241747294 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello2b\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"hello2b\"],\"resources\":{}}}" version="&Version{Version:v3.2.9,BuildDate:2022-03-02T21:41:01Z,GitCommit:ce91d7b1d0115d5c73f6472dca03ddf5cc2c98f4,GitTag:v3.2.9,GitTreeState:clean,GoVersion:go1.16.14,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:55.725Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:01.727Z" level=info msg="Main container completed"
time="2022-10-16T09:31:01.727Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:01.727Z" level=info msg="No output parameters"
time="2022-10-16T09:31:01.727Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:01.727Z" level=info msg="Killing sidecars []"
time="2022-10-16T09:31:01.727Z" level=info msg="Alloc=4184 TotalAlloc=9399 Sys=73809 NumGC=3 Goroutines=6"
time="2022-10-16T09:31:05.961Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.2.9
time="2022-10-16T09:31:05.966Z" level=info msg="Creating a emissary executor"
time="2022-10-16T09:31:05.966Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-3622003880 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello2a\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello2a\"],\"resources\":{}}}" version="&Version{Version:v3.2.9,BuildDate:2022-03-02T21:41:01Z,GitCommit:ce91d7b1d0115d5c73f6472dca03ddf5cc2c98f4,GitTag:v3.2.9,GitTreeState:clean,GoVersion:go1.16.14,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:31:05.966Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:06.966Z" level=info msg="Main container completed"
time="2022-10-16T09:31:06.967Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:06.967Z" level=info msg="No output parameters"
time="2022-10-16T09:31:06.967Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:06.967Z" level=info msg="Killing sidecars []"
time="2022-10-16T09:31:06.967Z" level=info msg="Alloc=4838 TotalAlloc=9437 Sys=74065 NumGC=3 Goroutines=6"
time="2022-10-16T09:30:44.478Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.2.9
time="2022-10-16T09:30:44.483Z" level=info msg="Creating a emissary executor"
time="2022-10-16T09:30:44.483Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-983838386 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello1\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"hello1\"],\"resources\":{}}}" version="&Version{Version:v3.2.9,BuildDate:2022-03-02T21:41:01Z,GitCommit:ce91d7b1d0115d5c73f6472dca03ddf5cc2c98f4,GitTag:v3.2.9,GitTreeState:clean,GoVersion:go1.16.14,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:44.483Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:30:45.483Z" level=info msg="Main container completed"
time="2022-10-16T09:30:45.483Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:30:45.483Z" level=info msg="No output parameters"
time="2022-10-16T09:30:45.484Z" level=info msg="No output artifacts"
time="2022-10-16T09:30:45.484Z" level=info msg="Killing sidecars []"
time="2022-10-16T09:30:45.484Z" level=info msg="Alloc=5006 TotalAlloc=9548 Sys=74577 NumGC=3 Goroutines=6"
time="2022-10-16T09:30:55.728Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-10-16T09:30:55.729Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-whalesay-template-2600658835 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello1\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello1\"],\"resources\":{}}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:30:55.729Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:02.730Z" level=info msg="Main container completed" error=""
time="2022-10-16T09:31:02.730Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:02.730Z" level=info msg="No output parameters"
time="2022-10-16T09:31:02.730Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:02.730Z" level=info msg="Deadline monitor stopped"
time="2022-10-16T09:31:02.730Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-10-16T09:31:02.730Z" level=info msg="Alloc=6111 TotalAlloc=12057 Sys=19666 NumGC=4 Goroutines=7"
time="2022-10-16T09:31:05.967Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-10-16T09:31:05.970Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-10-16T09:31:05.970Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-main podName=workflow-template-steps-p5l2l-whalesay-template-3622003880 template="{\"name\":\"whalesay-template\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"inner-hello2a\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"inner-hello2a\"],\"resources\":{}}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-10-16T09:31:05.970Z" level=info msg="Starting deadline monitor"
time="2022-10-16T09:31:07.972Z" level=info msg="Main container completed" error=""
time="2022-10-16T09:31:07.972Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-10-16T09:31:07.972Z" level=info msg="No output parameters"
time="2022-10-16T09:31:07.972Z" level=info msg="No output artifacts"
time="2022-10-16T09:31:07.972Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-10-16T09:31:07.972Z" level=info msg="Alloc=6859 TotalAlloc=12225 Sys=19922 NumGC=4 Goroutines=6"
```

Start with the supplied Sensor manifest and workflow-controller logs, reproducing the failure on v3.2.9 with the Azure EventHub trigger. Trace the transition where the two parallel pods are created and the `expected < 2 pods, got 2` error appears. Done means the reproducible workflow completes without that controller error.

Written by the indexing model from the issue text.