The three-node cluster failed to fully utilize the NIC bandwidth

We deployed a three-node cluster, where one node serves as the client and metadata server, while the other two nodes function as storage nodes. We configured the system with a dual-replication strategy, set the chunk to 1MB, and the stripe to 16. We utilized the USRBIO FIO with the following parameters: -numjobs=128 -fallocate=none -iodepth=256 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -direct=1 -rw=read -bs=1M --group_reporting -size=500M -mountpoint=/3fs/stage. However, neither the CPU, DRAM, NIC, nor the SSDs on the client/metadata node and storage nodes reached their maximum utilization. The 100Gbps NIC only achieved an actual throughput of approximately 40Gbps. Is this issue attributable to suboptimal cluster deployment methods, inadequate 3FS configurations, or improper fio settings?

Start with the reported three-node topology, dual-replication, 1MB chunk, 16 stripe, and the USRBIO FIO parameters. Compare the deployment and 3FS configuration with the workload settings, then identify which factor explains the roughly 40Gbps result. Done means documenting a reproducible cause and the configuration or workload evidence supporting it.

Written by the indexing model from the issue text.