[BUG]Issue with Zero Optimization for Llama-2-7b Fine-Tuning on Intel GPUs

Describe the bug
I’m experiencing an issue when fine-tuning the Llama-2-7b model from Hugging Face with Zero optimization enabled. I am running on 8 Intel Max 1550 GPUs using the code from the examples provided in Intel Extension for DeepSpeed.

The model loads and runs successfully without Zero optimization, but when I enable Zero optimization (particularly with stage 3), I encounter the following errors:
[rank0]: RuntimeError: could not create an engine
2024:11:05-02:39:09:(678567) |CCL_INFO| finalizing level-zero
2024:11:05-02:39:09:(678567) |CCL_INFO| finalized level-zero
0%| | 0/50 [00:00<?, ?it/s]
2024:11:05-02:39:09:(678572) |CCL_INFO| finalizing level-zero
2024:11:05-02:39:09:(678566) |CCL_INFO| finalizing level-zero
...
[2024-11-05 02:39:10,447] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 678572

**System info **
Model: Llama-2-7b from Hugging Face
GPUs: 8x Intel Max 1550 GPUs
Software:
• Intel Extension for pytorch
• DeepSpeed with Zero Optimization (Stage 3)
• oneCCL for communication backend

Launcher context
cd transformers
deepspeed --num_gpus=8 examples/pytorch/language-modeling/run_clm.py
--deepspeed tests/deepspeed/ds_config_zero3.json
--model_name_or_path meta-llama/Llama-2-7b-hf
--dataset_name wikitext
--dataset_config_name wikitext-2-raw-v1
--dataloader_num_workers 0
--per_device_train_batch_size 1
--warmup_steps 10
--max_steps 50
--bf16
--do_train
--output_dir /tmp/test-clm
--overwrite_output_dir

Reproduce the reported command from examples/pytorch/language-modeling/run_clm.py with tests/deepspeed/ds_config_zero3.json on the stated Intel GPU setup. Start with the Zero Stage 3 configuration and the “could not create an engine” failure, then use the surrounding launcher and runtime logs to narrow the cause. Done means the reported fine-tuning run completes with Zero optimization enabled.