Occasional failure to create s3 client when loading artifacts

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

Expecting artifacts to load consistently in the pod. However we notice at least 1 failure a day randomly.

```
Error (exit code 1): artifact metadata failed to load: failed to create new S3 client: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors
```

### Version

v3.3.9

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Any workflow pod that loads data from a store, s3 in our case.

It could be the internal library that's causing issues as well
https://github.com/aws/aws-sdk-go/issues/2914
https://github.com/aws-observability/aws-otel-collector/issues/1286

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Will add these logs
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Will add these logs
```

Start with the workflow-controller and wait-container logs described in the issue, then investigate the linked AWS SDK credential-provider issue while reproducing artifact loading from S3. The issue is done when the intermittent NoCredentialProviders failure is understood and artifact loading is consistently reliable, with any needed verification documented.

Written by the indexing model from the issue text.