lablup / lablup/backend.ai

Enable `CpusetMems` when CPU allocation is NUMA-local

Open
#11,217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Parent epic: #11216

## Main idea

`src/ai/backend/agent/docker/intrinsic.py:429–442` — the `CpusetMems` host-config key is commented out, so container memory is not pinned to the same NUMA node as the allocated CPU cores. NUMA tracking itself is fully implemented: `AffinityMap` (`src/ai/backend/agent/affinity_map.py:145–160`) already computes the distance matrix, and `CPUPlugin.list_devices()` (`intrinsic.py:213`) records a `numa_node` per core.

Gate `CpusetMems` on the allocated cores belonging to a single NUMA node, and set it to that node ID. Fall back to the current behavior (omit the key) when the allocation spans multiple nodes.

```python
# intrinsic.py, around line 434
numa_nodes = {core.numa_node for core in allocated_cores}
host_config = {"Cpus": len(cores), "CpusetCpus": ",".join(sorted_core_ids)}
if len(numa_nodes) == 1:
host_config["CpusetMems"] = str(next(iter(numa_nodes)))
```

Expected 2–3× improvement on remote-memory access latency for NUMA hosts when the allocation already happens to be node-local (which `AllocationStrategy.FILL` prefers today — see comment at `resources.py:176`).

## Alternative ideas

- Allow multi-node pinning (`CpusetMems="0,1"`) when the allocation spans exactly the selected set. Lower win, wider applicability.
- Always emit `CpusetMems` equal to the union of all nodes covered by allocated cores — functionally equivalent to the current default on most systems, so low value.

## Anything else?

Needs verification on a NUMA host. Single-socket machines are unaffected (single node = current behavior).

JIRA Issue: BA-5857

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.