alibaba / alibaba/clusterdata

Resource usage for task can be higher than resources requested

Open
#31 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.2k
Forks
482
PR merge metrics
No merged PRs in 30d

Description

There are many tasks in the dataset that utilize more resources than what was requested.

For instance, job_id:10771 task_id:66551 has plan_cpu:0.75 [1] from the following entry in _batch_task.csv_:
> 6301,6352,10771,66551,137,Terminated,**75**,0.01600704061294748

However, this task utilizes 7.66 (Max) and 0.99 (average) CPU as can be seen in batch_instance.csv:
> 6302,6339,10771,66551,427,Terminated,1,1,**7.66,0.99**,0.019309916392721248,0.012926772448424922

Can you clarify if the amount of resources used by tasks can be higher than the amount of resources requested? If not, what can explain these numbers?

Can I interpret the amount of resources requested as resources allocated by the scheduler?

[1] I divided the _plan_cpu_ value by 100 as explained in this issue: https://github.com/alibaba/clusterdata/issues/11

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the referenced entries in _batch_task.csv_ and batch_instance.csv, then review issue #11 for the plan_cpu interpretation. Compare the requested and observed CPU fields and trace any available dataset documentation or notebooks that define them. Done means documenting whether requested resources represent scheduler allocation and explaining the discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
data
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.