[BUG?] zero++ init problem

Describe the bug
Is it necessary to use assert in this line? (https://github.com/deepspeedai/DeepSpeed/blob/8577bd244903f454e319953a6ae1c3838558ac69/deepspeed/utils/groups.py#L663)

In the world of VLM, there are many cases that I have to initialize two different models (vision encoder & LLM, like this one https://github.com/LLaVA-VL/LLaVA-NeXT)

I do not understand full logic inside zero++, but it would be nicer if it shows just warning, not constrain it.
Any problem that could occur if we relieve this condition?

To Reproduce
load two huggingface model with from_pretrained with zero++ (zero_hpz_partition_size>1)

Launcher context
Accelerate