[Question] bf16 suppot DeepSpeedCPUAdamOptimizer and performance degradation with multiple instances

Hi, I'm trying out DeepSpeedCPUAdamOptimizer and I have a few questions about it.

During my testing, I discovered that DeepSpeedCPUAdamOptimizer does not seem to support bf16. Whenever I run the program, I consistently encounter the error message "interrupted by signal 11: SIGSEGV." Is this behavior expected? Are there any plans to support bf16 in the future?

I also encountered a performance issue when I replaced PyTorch Adam with DeepSpeedCPUAdamOptimizer in my training scripts. The performance seems to be degraded compared to the test script. After inspecting the source code, I noticed that #pragma omp parallel for is used to enable multithreading. However, in my setup where I have multiple GPUs running on a single node, multiple instances of optimizers may compete for CPU resources. Is there any workaround for this issue?

Start with the DeepSpeedCPUAdamOptimizer implementation, especially the bf16 handling and the #pragma omp parallel for path mentioned in the issue. Reproduce the SIGSEGV with bf16 and benchmark multiple optimizer instances across GPUs against the test script. Done means the support status and performance cause or workaround are established and documented.