[BUG] instruction/information about the max model size can be trained given a compute budget

Describe the bug

I found it is very difficult to know the max capability of DeepSpeed. I think DeepSpeed has great potential but it's a bummer that users poorly understand how to make use of it.

I personally want to fine-tune a 30B+ model (full model, not lora) on 8~16 A100 (40GB) gpus. I tried to run it following some tutorials (DeepSpeed Integration and How to use DeepSpeed by huggingface , because official docs are so scarce and offer little help.

The current methodology is just trial and error and it is very inefficient (my current progress is it is doable up to 16B with ZeRO stage 3 and activation checkpointing enabled, but not 30B yet).

I wonder if the team can provide some rule of thumb that (1) given a specific compute budget (say 16 * A100 40GB) quickly determine if it is trainable with DeepSpeed; (2) what hyperparameters are mostly effective to reduce gpu memory usage (allgather_bucket_size, stage3_max_live_parameters, stage3_max_reuse_distance, sub_group_size etc.)

Thanks,
Rui

Start with the linked Hugging Face DeepSpeed Integration and Accelerate guides, then review the issue's ZeRO Stage 3, activation checkpointing, and memory-related parameters. Done means documenting a reliable way to estimate trainable model size from GPU budget and explaining which parameters reduce memory usage, with examples for configurations such as 16 A100 40GB GPUs.