Data partitioning
Đang mở
Conformance
Epic
Menas
priority: high
Standardization
- Ngôn ngữ chính
- Scala
- Star
- 33
- Fork
- 16
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Background
Sometimes the data coming in into Enceladus are not well partitioned.
## Goal
Make Enceladus to output data files in effective sizes to well leverage partioning
## Expected Task List
A list of expected issues that will be needed to achieve this Epic
- investigate the problematic data
- create the library
- enhance dataset data model with `recordsPerPartition` value
- optional dataset setting
- heuristic based on anylysis to be saved
- implement partitioning by record count
- warning if partition size is over limit
- warning if record count is off
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.