Incompatibility between `plugin` and `http` templates

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

It seems that users cannot have `plugins` in their cluster and expect `http` templates to work.
This is both undocumented and unexpected.

# Setup
- I installed Argo Workflows from its official release manifest. Versions are documented below.
- I added the env var as directed [by the docs](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#configuration):
```yaml
- name: ARGO_EXECUTOR_PLUGINS
value: "true"
```
- I installed the `hello-world` [plugin from argoproj-labs](https://raw.githubusercontent.com/argoproj-labs/argo-workflows-hello-executor-plugin/refs/heads/main/hello-executor-plugin-configmap.yaml) into the same namespace as the controller.
- I ran a basic workflow with a plugin template to call that plugin. I see the plugin as a sidecar to that Pod.

## Testing on 3.6.4
- If ARGO_EXECUTOR_PLUGINS == `true` and there IS a plugin installed, the plugin is added as a sidecar of the http template pod AND the http template fails with unclear reasoning. << This is unexpected and undocumented.
- Even more fun is I have managed to make this pass on occasion. It's intermittent
- If ARGO_EXECUTOR_PLUGINS == `true` but there is no plugin installed in the cluster, http template runs fine.
- If ARGO_EXECUTOR_PLUGINS is not set and there is no plugin installed in the cluster, http template runs fine.

## Testing on 3.5.14
Initially, nothing worked and then I saw extra logs in the controller that aren't on 3.6.x:
```
time="2025-02-20T13:29:59.644Z" level=info msg="Updated message -> failed to check if secret argo-workflows-agent-ca-certificates exists: secrets \"argo-workflows-agent-ca-certificates\" is forbidden: User \"system:serviceaccount:argo:argo\" cannot get resource \"secrets\" in API group \"\" in the namespace \"workflows\"" namespace=workflows workflow=http-jnn24
```
(`argo` is the namespace and serviceaccount of the controller, not the running workflow)

After addressing this with extra RBAC, my findings were the same as 3.6.4:
- If ARGO_EXECUTOR_PLUGINS == `true` and there IS a plugin installed, the plugin is added as a sidecar of the http template pod AND the http template fails with unclear reasoning. << This is unexpected and undocumented.
- Even more fun is I have managed to make this pass on occasion. It's intermittent
- If ARGO_EXECUTOR_PLUGINS == `true` but there is no plugin installed in the cluster, http template runs fine.
- If ARGO_EXECUTOR_PLUGINS is not set and there is no plugin installed in the cluster, http template runs fine.

## Additional information
I also repeated the 3.6.4 tests with that new secrets permission added, but the results did not change.

## Summary of issues
1. I can't run http templates if there are plugins in my cluster. At a mimum the documentation needs to be updated to reflect this, but it feels like a bug.
1. This shouldn't be intermittent. This screams "bug" to me.
1. 3.5 needed more permissions. It's not clear why these permissions are required. An explanation is required here.
1. I see no reason why a plugin should run when an http template is called.

### I have not tested on `latest`

### Version(s)

v3.6.4, v3.5.14

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: http-
namespace: workflows
annotations:
workflows.argoproj.io/description: >-
Demonstrates the HTTP template that executes a HTTP request.
workflows.argoproj.io/maintainer: 'Pipekit Inc'
spec:
serviceAccountName: workflows
entrypoint: main
templates:
- name: main
steps:
- - name: get-example-homepage
template: http
- name: http
http:
timeoutSeconds: 20
url: https://example.com/
method: "GET"
```

### Logs from the workflow controller

```text
This is from when plugins=true and there is a plugin in the cluster on 3.6.4:

time="2025-02-20T14:09:21.803Z" level=info msg="Processing workflow" Phase= ResourceVersion=810 namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.810Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=http-jv7f6
time="2025-02-20T14:09:21.810Z" level=info msg="Updated phase -> Running" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="Retry node http-jv7f6 initialized Running" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="Steps node http-jv7f6-2014232316 initialized Running" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="StepGroup node http-jv7f6-2592250198 initialized Running" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.811Z" level=info msg="Retry node http-jv7f6-438562091 initialized Running" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.812Z" level=info msg="HTTP node http-jv7f6-679105702 initialized Pending" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.812Z" level=info msg="Workflow step group node http-jv7f6-2592250198 not yet completed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.812Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.812Z" level=info msg="Creating TaskSet" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.820Z" level=info msg=reconcileAgentPod namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.826Z" level=warning msg="couldn't retrieve node for nodeName , will get nil templateDeadline" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.826Z" level=warning msg="couldn't get boundaryTemplate through nodeName " namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.837Z" level=info msg="Created Agent pod" namespace=workflows podName=http-jv7f6-1340600742-agent workflow=http-jv7f6
time="2025-02-20T14:09:21.837Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:21.837Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:21.862Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=817 workflow=http-jv7f6
time="2025-02-20T14:09:22.839Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=817 namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.840Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=http-jv7f6
time="2025-02-20T14:09:22.840Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.840Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:22.840Z" level=info msg="Workflow step group node http-jv7f6-2592250198 not yet completed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.840Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.840Z" level=info msg="Creating TaskSet" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.845Z" level=info msg=reconcileAgentPod namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.845Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:22.845Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:23.847Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=817 namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.847Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=http-jv7f6
time="2025-02-20T14:09:23.847Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.847Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:23.848Z" level=info msg="Workflow step group node http-jv7f6-2592250198 not yet completed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.848Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.848Z" level=info msg="Creating TaskSet" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.852Z" level=info msg=reconcileAgentPod namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.852Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:23.852Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:25.123Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=817 namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.123Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=http-jv7f6
time="2025-02-20T14:09:25.123Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.123Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:25.123Z" level=info msg="Workflow step group node http-jv7f6-2592250198 not yet completed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.123Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.123Z" level=info msg="Creating TaskSet" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.128Z" level=info msg=reconcileAgentPod namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.128Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:25.128Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:25.136Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=840 workflow=http-jv7f6
time="2025-02-20T14:09:26.132Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=840 namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.133Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=http-jv7f6
time="2025-02-20T14:09:26.133Z" level=info msg=updateAgentPodStatus namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.133Z" level=info msg=assessAgentPodStatus namespace=workflows podName=http-jv7f6-1340600742-agent
time="2025-02-20T14:09:26.134Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="Node not set to be retried after status: Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-438562091 phase Running -> Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-438562091 message: context canceled" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-438562091 finished: 2025-02-20 14:09:26.134683921 +0000 UTC" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="Step group node http-jv7f6-2592250198 deemed failed: child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2592250198 phase Running -> Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2592250198 message: child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2592250198 finished: 2025-02-20 14:09:26.134759254 +0000 UTC" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="step group http-jv7f6-2592250198 was unsuccessful: child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="Outbound nodes of http-jv7f6-438562091 is [http-jv7f6-679105702]" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="Outbound nodes of http-jv7f6-2014232316 is [http-jv7f6-679105702]" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2014232316 phase Running -> Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2014232316 message: child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.134Z" level=info msg="node http-jv7f6-2014232316 finished: 2025-02-20 14:09:26.134836837 +0000 UTC" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="Node not set to be retried after status: Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="node http-jv7f6 phase Running -> Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="node http-jv7f6 message: child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="node http-jv7f6 finished: 2025-02-20 14:09:26.135065962 +0000 UTC" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg=reconcileAgentPod namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="Updated phase Running -> Failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="Updated message -> child 'http-jv7f6-438562091' failed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.135Z" level=info msg="Marking workflow completed" namespace=workflows workflow=http-jv7f6
time="2025-02-20T14:09:26.143Z" level=info msg="cleaning up pod" action=deletePod key=workflows/http-jv7f6-1340600742-agent/deletePod
time="2025-02-20T14:09:26.162Z" level=info msg="Queueing Failed workflow workflows/http-jv7f6 for delete in 1h0m0s due to TTL"
time="2025-02-20T14:09:26.162Z" level=info msg="Workflow update successful" namespace=workflows phase=Failed resourceVersion=846 workflow=http-jv7f6
```

### Logs from in your workflow's wait container

```text
There is no `wait` container.
```

Start with the executor_plugins configuration documentation and reproduce the supplied minimal HTTP workflow with ARGO_EXECUTOR_PLUGINS enabled and the hello-world plugin installed. Compare the controller logs and behavior across the reported versions and configurations; this is done when the plugin and HTTP-template interaction is either corrected or clearly documented, including the required permissions and expected failure behavior.

Written by the indexing model from the issue text.