lablup / lablup/backend.ai

Expand NUMA-aware resource allocation feature to finer scope

Open
#2,007 0 comments 0 reactions 1 assignee Claimed by @achimnol View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
14h 15m
Merged PRs (30d)
335

Description

## Main idea

Follow-up of #491.It will be great to update our NUMA aware resource allocation feature, such as taking accelerator's parent PCIe switch into account, for example.

### (Agent) Expansion of [AffinityMap](https://github.com/lablup/backend.ai/blob/02aad902df84c1cf71c9808d615af5d79f66e383/src/ai/backend/agent/affinity_map.py)

Currently the device distance calculation is based on the PCI-reported numa node index only.

We could extend this distance calculation logic to consider additional information from the PCIe bus address. For example, we could use the bus number to further distinguish PCIe switches.

First, we need to check if we can determine the existence and layout of PCIe switches from the bus addresses.

- ref: https://www.linkedin.com/pulse/demystifying-pcie-addressing-understanding-bus-device-dipan-patel/
### (Manager) Hierarchical agent selection strategy for multi-node sessions

This will be a follow-up of #1394 and #1655.

It will include additional location metadata in agent heartbeats (e.g., rack number, rack groups, etc.) and considering them when distributing the containers of a single cluster session.

JIRA Issue: BA-132

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.