InternLM / InternLM/AcmeTrace

Kalos cluster under-utilised

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
183
Forks
11
PR merge metrics
No merged PRs in 30d

Description

Hi, when I explored the cluster utilisation rate (number of running GPUs / total number of GPUs) based on the job start time, end time, and the number of GPUs for each job, I found that the maximum utilisation rate of the Kalos cluster is only around 70%, and there are lots of periods where less than 40% or even 20% of the total GPUs of the cluster are used, which is quite weird and is not the case for Seren. I also found that the Seren data has ~800k job records, while Kalos only has ~60k. Does this mean that not all jobs are recorded for Kalos, which further leads to the severe under-utilisation?

Sincerely appreciate it if you could help clarify this. Also thank you so much for sharing this fantastic dataset.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.