Intel MPI Benchmarks (IMB-MPI1) performance issue? with EFA and Rocky Linux 8 custom image

**Required Info:**
- AWS ParallelCluster version [e.g. 3.1.1]: 3.8.0
- Full cluster configuration without any credentials or personal data.
- cluster configuration https://rpa.st/AAJQ
- whole procedure https://ciq.com/blog/how-to-use-aws-parallelcluster-3-8-0-with-rocky-linux-8/
- Cluster name: rocky8-cluster
- Output of `pcluster describe-cluster` command.
```
{
"creationTime": "2023-12-20T16:19:01.897Z",
"headNode": {
"launchTime": "2023-12-20T16:23:42.000Z",
"instanceId": "i-******",
"publicIpAddress": "***.***.***.***",
"instanceType": "t2.xlarge",
"state": "running",
"privateIpAddress": "10.0.0.230"
},
"version": "3.8.0",
"clusterConfiguration": {
"url": "******"
},
"tags": [
{
"value": "3.8.0",
"key": "parallelcluster:version"
},
{
"value": "rocky8-cluster",
"key": "parallelcluster:cluster-name"
}
],
"cloudFormationStackStatus": "CREATE_COMPLETE",
"clusterName": "rocky8-cluster",
"computeFleetStatus": "RUNNING",
"cloudformationStackArn": "arn:aws:cloudformation:ap-northeast-1:******:stack/rocky8-cluster/******",
"lastUpdatedTime": "2023-12-20T16:19:01.897Z",
"region": "ap-northeast-1",
"clusterStatus": "CREATE_COMPLETE",
"scheduler": {
"type": "slurm"
}
}
```
- [Optional] Arn of the cluster CloudFormation main stack:

**Bug description and how to reproduce:**

- Issue
- Intel MPI Benchmark `IMB-MPI1 PingPong` performance with EFA looks not as expected
- It looks like EFA was used according to the log attached to this ticket
- I would like to confirm if this is the expected range of performance.
- [[result 1](https://rpa.st/AGWQ)] with `FI_PROVIDER=EFA FI_LOG_LEVEL=Debug srun --mpi=pmix ~/mpi-benchmarks/src_c/IMB-MPI1 -msglog 3:28 PingPong`
- [[result 2](https://rpa.st/QDNA)] with `FI_PROVIDER=EFA FI_LOG_LEVEL=Debug mpirun --mca pml cm --mca mtl ofi ~/mpi-benchmarks/src_c/IMB-MPI1 -msglog 3:28 PingPong`
```
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
268435456 1 99276.95 2703.91
```

- The steps to reproduce the behavior.
- Build Intel MPI Benchmark
- ```
cd ~/
git clone https://github.com/intel/mpi-benchmarks.git
cd mpi-benchmarks/src_c
make all
```
- Submit job. Example job script `job.sh` https://rpa.st/WBZQ
- Output `slurm-*.out` https://rpa.st/22UA