Simple Example for Single-GPU Simple Model Training & Stand-Alone DeepSpeedCPUAdam

First of all, thanks for such an amazing project. I am interested in using the ZeRO-Offload training functionality due to its speed and memory advantages. I have managed to get it work using the examples in the tutorial but I was only able to fit the half of my original batch-size. I will later follow-up with my code snippet and configuration details to see what I have done wrong.

(1) It would be great if there was a simple model training without multi-gpu or complex language models or GANs.
(2) It would be great if there could be away of directly using DeepSpeedCPUAdam as a stand-alone optimizer; if that is sufficient to get those speed and memory advantages on single-gpu model training.

The issue points to the ZeRO-Offload tutorial examples and DeepSpeedCPUAdam; start by reviewing those examples and the optimizer's current usage. Done would mean a simple single-GPU model-training example without multi-GPU or complex models, plus a documented way to use DeepSpeedCPUAdam standalone; follow-up configuration details may still be needed.