alibaba / alibaba/clusterdata

Question about batch jobs input

Open
#67 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.2k
Forks
482
PR merge metrics
No merged PRs in 30d

Description

Hello,

In MR, Spark, we are assuming each mapper or reducer handles portion of data. The data size for each map or reduce instance is at most equal to hdfs block size. So the input size can be estimated by the multipication of of blocksize*number of instances. Is the same assumption hold with Alibaba trace ?

Thank you for releasing these traces. The DAG information is so valuable. It would be great if it could include, the inputs to each job and the input sizes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.