Is GPU throughput reasonable?

I currently have some tests on Zero3 infinite and have had some problems and would like your help.

Machine configuration: two nodes, each node a piece of A100-PCIE-40GB, RAM 126G (actual operation available 60G), SSD 1TB (Samsung 980)

Benchmark Code：/DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3/

Model cases tested：
HIDDEN_SIZE / NUM_ATTN_HEADS/ NUM_LAYERS/ BATCHSIZE = 4096/16/50/8 （Model size 10B）
GPU memory occupies 13395/40537MB
RAM occupancy 109/126G, (60G at idle)
80G of swap files stored in nvme file system
Effective Tera Flops per GPU is 1.5TFLPOS

Question：
Whether the GPU throughput achieved under the current environment configuration is reasonable, and whether the throughput can be increased by increasing the batch size or other configurations
Effective Tera Flops per GPU calculated in flops_calculator of DeepSpeedExamples is about 1.5 TFLPOS. But deepspeed profile tested FLOPS per GPU is 2.32 GFLOPS.(deepspeed _profile.txt is generated by deepspeed profile and train.log is the information output during training)

deepspeed _profile.txt
train.log

I hope to get your help, thank you very much!

Start with the DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3 benchmark configuration and inspect the attached deepspeed _profile.txt and train.log. Compare the reported FLOPS and throughput measurements for the stated two-node A100 setup; done means explaining the discrepancy and identifying configuration changes that could increase throughput.