[BUG] Using Zero++, evaluation loss is high and evaluate accuracy is always 0

Describe the bug
Using Zero++, evaluation loss is high and evaluate accuracy is always 0

To Reproduce
Steps to reproduce the behavior:

1. Use this deepspeed config
   "zero_optimization": {
   "stage": 3,
   "offload_param": {
   "device": "cpu",
   "pin_memory": true
   },
   "offload_optimizer": {
   "device": "cpu",
   "pin_memory": true
   },
   "overlap_comm": true,
   "contiguous_gradients": true,
   "sub_group_size": 1e9,
   "reduce_bucket_size": "auto",
   "zero_hpz_partition_size": 8,
   "zero_quantized_gradients": true,
   "stage3_prefetch_bucket_size": "auto",
   "stage3_param_persistence_threshold": "auto",
   "stage3_max_live_parameters": 1e9,
   "stage3_max_reuse_distance": 1e9,
   "stage3_gather_16bit_weights_on_model_save": true
   },

2. Use zero++ to fine tune falcon40-B, train it for a few hundred steps.

3. Got such evaluation result
   INFO Evaluation results: {
   "eval_loss": 2.1119680404663086,
   "eval_accuracy": 0.0,}

Expected behavior
Got a low loss and non-zero accuracy

ds_report output
get 0 accuracy after fine-tuning.
Without fine-tuning, the accuracy is more than 0.

Screenshots

System info (please complete the following information):

• GPU count and types [1*8 A100 GPUS, with memory 80G]
• Interconnects (if applicable) 1 node.
• Python version 2.*

Launcher context
Use torchrun

No source file or test is named. Start by reproducing the Falcon40-B fine-tuning run with the supplied Zero++ configuration under torchrun, then compare evaluation before and after fine-tuning and inspect the evaluation output. Done means evaluation loss and accuracy behave as expected rather than producing consistently zero accuracy.