[Question] Is CPU-only inference supported ?

I set ds_accelerator to cpu,
and pip install intel_extension_for_pytorch, setup torch_ccl from https://github.com/intel/torch-ccl .
I run a llama2-7B model.

And from the source code, I find that pre_norm_rms has only CUDA support.
So, I wonder that if the deepspeed can run in CPU-only inference environment ?

[2023-11-28 18:48:08,800] [INFO] [engine_v2.py:82:__init__] Building model...
Traceback (most recent call last):
  File "examples/test.py", line 18, in <module>
    pipe = pipeline("Llama-2-7b-chat-ms")
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/mii/api.py", line 159, in pipeline
    inference_engine = load_model(model_config)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/mii/modeling/models.py", line 20, in load_model
    inference_engine = build_hf_engine(
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/engine_factory.py", line 106, in build_hf_engine
    return InferenceEngineV2(policy, engine_config)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/engine_v2.py", line 83, in __init__    self._model = self._policy.build_model(self._config, self._base_mp_group)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/model_implementations/inference_policy_base.py", line 156, in build_model
    self.model = self.instantiate_model(engine_config, mp_group)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/model_implementations/llama_v2/llama_v2_policy.py", line 17, in instantiate_model
    return Llama2InferenceModel(config=self._model_config, engine_config=engine_config, base_mp_group=mp_group)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/model_implementations/inference_transformer_base.py", line 206, in __init__
    self.make_norm_layer()
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/model_implementations/inference_transformer_base.py", line 504, in make_norm_layer
    self.norm = heuristics.instantiate_pre_norm(norm_config, self._engine_config)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/modules/heuristics.py", line 160, in instantiate_pre_norm
    return DSPreNormRegistry.instantiate_config(config)
  File ".conda/envs/deepspeed-cpu/lib/python3.8/site-packages/deepspeed/inference/v2/modules/module_registry.py", line 38, in instantiate_config
    raise ValueError(f"Config {config_bundle.config} is not supported by {target_implementation}")
ValueError: Config max_tokens=768 type='rms_norm' channels=4096 residual_dtype=torch.float16 input_dtype=torch.float16 output_dtype=torch.float16 eps=1e-05 is not supported by <class 'deepspeed.inference.v2.modules.implementations.pre_norm.cuda_pre_rms.DSPreRMSCUDAModule'>
[2023-11-28 18:48:10,264] [INFO] [launch.py:320:sigkill_handler] Killing subprocess 28105

Start with deepspeed/inference/v2/modules/heuristics.py and the pre-norm implementations referenced by the traceback, then inspect the Llama v2 path in deepspeed/inference/v2/model_implementations/llama_v2/llama_v2_policy.py. Reproduce the CPU-only Llama-2 inference setup and determine the scope of CPU support; done would require a documented support decision and a passing CPU inference path if support is added.