EMA Model Save Fails Due to CPU Memory Exhaustion

I can save the model ckpt after fine-tuning it on 2 A100 GPUs with --cpu_offload True and --sharding_strategy FULL_SHARD.
But the EMA model can't be saved correctly (due to CPU memory exhaustion, ~433 GB) even I consider:
with FSDP.state_dict_type(
ema_model,
StateDictType.SHARDED_STATE_DICT,
ShardedStateDictConfig(offload_to_cpu=True),
):
...

Have anyone encounter this?
I'd appreciate your help!

Reproduce EMA checkpoint saving with 2 A100 GPUs, --cpu_offload True, --sharding_strategy FULL_SHARD, and SHARDED_STATE_DICT with offload_to_cpu enabled. Trace the EMA model's FSDP state-dict path and compare it with the working model checkpoint; done means the EMA checkpoint saves without exhausting roughly 433 GB of CPU memory.

Written by the indexing model from the issue text.