Single step of sibling step groups that uses locks do not release lock when workflow is stopped

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [X] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

When submitting the job provided in this report, I expect to be able to stop the job at any time and have any acquired locks to be released.

When submitting the job provided in this report, acquired locks are not released from the workflow in situations where subsequent siblings have not yet received a lock.

### Setup

In configmap, set "test-scale: N" (we only want a few siblings to run at the same time to reproduce this bug)

### My insights so far

1. This happens specifically in cases where there are other `process-one-step-1`s within this same workflow that have not yet acquired a lock.
2. When I click Stop, the locks are released by the current holders, but the subsequent holders do not relinquish them. These new lock holders do not actually run, and are marked as Successful. (I know who is holding the lock because I look in the Workflow directly)
3. I'm 99% sure this has to do with Steps templates. This is because if I remove `process-one`, and point to `process-one-step-1` after `create-ids`s is run, behavior is as expected.

### Version

v3.4.6

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: lock-bug
namespace: argo
spec:
entrypoint: entrypoint
templates:
- name: entrypoint
steps:
- - name: create-ids
template: create-ids
- - name: process-one
template: process-one
arguments:
parameters:
- name: id
value: '{{item}}'
withParam: '{{steps.create-ids.outputs.result}}'
- name: process-one
inputs:
parameters:
- name: id
steps:
- - name: process-one-step-1
template: process-one-step-1
arguments:
parameters:
- name: id
value: '{{inputs.parameters.id}}'
- name: create-ids
script:
image: python:3.8.16-alpine3.16
command:
- python
source: |
print("[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]")
- name: process-one-step-1
inputs:
parameters:
- name: id
script:
image: python:3.8.16-alpine3.16
command:
- python
source: >
import time
time.sleep(60)
synchronization:
semaphore:
configMapKeyRef:
name: scale-config
key: test-scale
serviceAccountName: argo
volumes:
- name: depot
persistentVolumeClaim:
claimName: depot-pvc
podGC:
strategy: OnPodSuccess
```

### Logs from the workflow controller

```text
time="2023-04-06T19:20:06.317Z" level=info msg="Workflow step group node lock-bug8rnxt-3917437900 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.319Z" level=info msg="Workflow step group node lock-bug8rnxt-3646440252 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.322Z" level=info msg="signaled container" container=main error="unable to upgrade connection: container not found (\"main\")" namespace=argo pod=lock-bug8rnxt-process-one-step-1-1178071149 stderr="" stdout=""
time="2023-04-06T19:20:06.322Z" level=info msg="https://10.100.0.1:443/api/v1/namespaces/argo/pods/lock-bug8rnxt-process-one-step-1-1178071149/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-04-06T19:20:06.323Z" level=info msg="signaled container" container=main error="unable to upgrade connection: container not found (\"main\")" namespace=argo pod=lock-bug8rnxt-process-one-step-1-3533322073 stderr="" stdout=""
time="2023-04-06T19:20:06.323Z" level=info msg="Workflow step group node lock-bug8rnxt-698770632 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.323Z" level=info msg="https://10.100.0.1:443/api/v1/namespaces/argo/pods/lock-bug8rnxt-process-one-step-1-3533322073/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-04-06T19:20:06.327Z" level=info msg="Workflow step group node lock-bug8rnxt-1448145272 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.331Z" level=info msg="Workflow step group node lock-bug8rnxt-617251268 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.335Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=argo pod=lock-bug8rnxt-process-one-step-1-2094510219 stderr="" stdout=""
time="2023-04-06T19:20:06.338Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=argo pod=lock-bug8rnxt-process-one-step-1-3533322073 stderr="" stdout=""
time="2023-04-06T19:20:06.349Z" level=info msg="Workflow step group node lock-bug8rnxt-1887021364 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.382Z" level=info msg="signaled container" container=wait error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"27f6bd394db156b8571b16298c96206d63e1c444998b25feffcac4f27e86dc75\": OCI runtime exec failed: exec failed: unable to start container process: read init-p: connection reset by peer: unknown" namespace=argo pod=lock-bug8rnxt-process-one-step-1-1178071149 stderr="" stdout=""
time="2023-04-06T19:20:06.401Z" level=info msg="signaled container" container=wait error="" namespace=argo pod=lock-bug8rnxt-process-one-step-1-3534690819 stderr= stdout="killing 1 with terminated\n"
time="2023-04-06T19:20:06.401Z" level=info msg="Workflow step group node lock-bug8rnxt-1297000069 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.401Z" level=info msg="Workflow step group node lock-bug8rnxt-2411759446 not yet completed" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.401Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.401Z" level=info msg=reconcileAgentPod namespace=argo workflow=lock-bug8rnxt
time="2023-04-06T19:20:06.415Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=669654504 workflow=lock-bug8rnxt
```

### Logs from in your workflow's wait container

```text
time="2023-04-06T19:20:05.762Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-04-06T19:20:06.026Z" level=info msg="Saving file to s3" bucket=sai-artifacts-dev-us-east-2 endpoint=s3.amazonaws.com key=lock-bug8rnxt/lock-bug8rnxt-process-one-step-1-3533322073/main.log path=/tmp/argo/outputs/logs/main.log
time="2023-04-06T19:20:06.059Z" level=info msg="Save artifact" artifactName=main-logs duration=297.015444ms error="" key=lock-bug8rnxt/lock-bug8rnxt-process-one-step-1-3533322073/main.log
time="2023-04-06T19:20:06.059Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-04-06T19:20:06.059Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-04-06T19:20:06.069Z" level=info msg="Create workflowtaskresults 201"
time="2023-04-06T19:20:06.070Z" level=info msg="Alloc=11671 TotalAlloc=18941 Sys=33746 NumGC=4 Goroutines=9"
time="2023-04-06T19:20:06.038Z" level=info msg="No output parameters"
time="2023-04-06T19:20:06.038Z" level=info msg="No output artifacts"
time="2023-04-06T19:20:06.038Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: lock-bug8rnxt/lock-bug8rnxt-process-one-step-1-3534690819/main.log"
time="2023-04-06T19:20:06.038Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-04-06T19:20:06.292Z" level=info msg="Saving file to s3" bucket=sai-artifacts-dev-us-east-2 endpoint=s3.amazonaws.com key=lock-bug8rnxt/lock-bug8rnxt-process-one-step-1-3534690819/main.log path=/tmp/argo/outputs/logs/main.log
time="2023-04-06T19:20:06.719Z" level=info msg="Save artifact" artifactName=main-logs duration=681.247335ms error="" key=lock-bug8rnxt/lock-bug8rnxt-process-one-step-1-3534690819/main.log
time="2023-04-06T19:20:06.720Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-04-06T19:20:06.720Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-04-06T19:20:06.727Z" level=info msg="Create workflowtaskresults 201"
time="2023-04-06T19:20:06.729Z" level=info msg="Alloc=10512 TotalAlloc=19253 Sys=34258 NumGC=4 Goroutines=9"
```

Start by running the YAML workflow reproducer with the scale-config ConfigMap and stopping it while sibling process-one-step-1 nodes are queued. Trace the workflow controller's Steps-template synchronization and lock reconciliation using the supplied controller and wait-container logs; done means stopping the workflow releases locks held by both running and not-yet-running sibling steps.

Written by the indexing model from the issue text.