[REQUEST] Auto-Tuning CPU Core Binding for DeepSpeed&ZenFlow

Description

Currently, DeepSpeed offers --bind_cores_to_rank and --bind_core_list flags to bind CPU cores, but these require explicit specification from the user. While core binding works, it is not fully automated and does not adapt dynamically to different NUMA node configurations or CPU layouts.

Problem

The current core binding functionality requires users to manually specify which cores to bind to each worker. This approach does not provide flexibility in handling complex CPU layouts, particularly when dealing with non-sequential core IDs or varying NUMA node configurations.

Proposed Discussion

We may explore the possibility of auto-tuning CPU core binding to find optimal strategies automatically. This would involve:

• Automatic Core Binding: Automatically determine the best core binding across NUMA nodes without requiring user input.

• Adaptation to Non-Sequential Core Layouts (due to logical cores): Account for irregularities in core numbering (e.g., non-sequential core IDs within a NUMA node) to ensure efficient binding.

• Maximizing Core Utilization: Ensure that the binding maximizes CPU utilization and minimizes memory contention across workers.

Possible Context

Deepspeed CPU binding: https://github.com/deepspeedai/DeepSpeed/blob/master/deepspeed/utils/numa.py#L117
ZenFlow CPU binding:
zenflow_optimizer_process() at #7391

Start with the CPU-binding implementation in deepspeed/utils/numa.py around line 117, then inspect the zenflow_optimizer_process() entry point referenced from issue #7391. Compare the existing bind_cores_to_rank and bind_core_list behavior with the proposed NUMA and non-sequential-core requirements; the work is done when an agreed auto-tuning design is implemented and its behavior is verified.