[AutoTP + DS2] no memory reduction when using auto = 4

I trained model qwen2.5-7B in GPU with llamafactory deepspeed zero2 + autotp. And there is no obvious memory reduction.

When individually using zero2, the average memory of 8 cards is 42g, while the average of zero2+autotp=4 is also nearly 42.

Here is my deepspeed config.

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "zero_allow_untested_optimizer": true,
  "fp16": {
    "enabled": "auto",
    "loss_scale": 0,
    "loss_scale_window": 1000,
    "initial_scale_power": 16,
    "hysteresis": 2,
    "min_loss_scale": 1
  },
  "bf16": {
    "enabled": "auto"
  },
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 5e8,
    "overlap_comm": false,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e8,
    "contiguous_gradients": true,
    "round_robin_gradients": true
  },
  "tensor_parallel":{
      "autotp_size": 4
  }
}

Start by reproducing the reported comparison with the supplied DeepSpeed configuration: ZeRO stage 2 alone versus AutoTP with autotp_size 4, using the same Qwen2.5-7B training setup. Measure per-GPU memory in both runs and trace the AutoTP and ZeRO integration to identify why memory remains near 42 GB. Done means the cause is isolated and the combined configuration shows the expected reduction or the limitation is documented.