AbsaOSS / AbsaOSS/enceladus

Data partitioning

未关闭
#1,961 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Conformance Epic Menas priority: high Standardization
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

## Background
Sometimes the data coming in into Enceladus are not well partitioned.

## Goal
Make Enceladus to output data files in effective sizes to well leverage partioning

## Expected Task List
A list of expected issues that will be needed to achieve this Epic
- investigate the problematic data
- create the library
- enhance dataset data model with `recordsPerPartition` value
- optional dataset setting
- heuristic based on anylysis to be saved
- implement partitioning by record count
- warning if partition size is over limit
- warning if record count is off

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。