Is GPU throughput reasonable?

I currently have some tests on Zero3 infinite and have had some problems and would like your help.

Machine configuration: two nodes, each node a piece of A100-PCIE-40GB, RAM 126G (actual operation available 60G), SSD 1TB (Samsung 980)

Benchmark Code：/DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3/

Model cases tested：

1. HIDDEN_SIZE / NUM_ATTN_HEADS/ NUM_LAYERS/ BATCHSIZE = 4096/16/50/8 （Model size 10B）
2. GPU memory occupies 13395/40537MB
3. RAM occupancy 109/126G, (60G at idle)
4. 80G of swap files stored in nvme file system
5. Effective Tera Flops per GPU is 1.5TFLPOS

Question：

1. Whether the GPU throughput achieved under the current environment configuration is reasonable, and whether the throughput can be increased by increasing the batch size or other configurations
2. Effective Tera Flops per GPU calculated in flops_calculator of DeepSpeedExamples is about 1.5 TFLPOS. But deepspeed profile tested FLOPS per GPU is 2.32 GFLOPS.(deepspeed _profile.txt is generated by deepspeed profile and train.log is the information output during training)

deepspeed _profile.txt
train.log

I hope to get your help, thank you very much!

Start with the DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3 benchmark configuration, then compare the flops_calculator result with deepspeed _profile.txt and train.log. Done means determining whether the reported GPU throughput is expected and clearly accounting for the difference between the two FLOPS measurements.