alibaba / alibaba/clusterdata

Question Regarding Normalized Memory Usage

Open
#61 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.2k
Forks
482
PR merge metrics
No merged PRs in 30d

Description

My question is in column `plan_mem` in `batch_task.csv`. I am still a little bit confused about the normalization standard. The schema mentions this column, plan_mem, specifies normalized memory requested for each instance of the task and it's `Normalized to the largest memory size of all machines`.

Few tasks have an even larger number than 15 in plan_mem column (e.g., 15.45, 17.17).
1. task_NDg2ODM2NDIyMDczNDQ4NzMzOA==,50,j_3003670,11,Terminated,233037,234282,700,17.17
2. task_NDg2ODM2NDIyMDczNDQ4NzMzOA==,50,j_3685110,11,Terminated,471423,493080,700,15.45

Does it mean each instance in this task take around 17.17 times of the `largest memory in a physical machine` in the whole memory? If I just assume the biggest memory on a physical machine in the cluster is 128GB, does each instance will take 2TB memory and the whole task will take 100TB memory eventually?

Thank you very much.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the schema entry for plan_mem and the cited rows in batch_task.csv; compare the stated normalization rule with the task and instance fields shown in the dataset. Done means the normalization semantics and the meaning of values above 1 are clearly documented in the issue or schema.

Written by the indexing model from the issue text.

Assessment

Domain
data, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.