AbsaOSS / AbsaOSS/cobrix

Optimization of input file split

未關閉
#585 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Scala
星號
170
分支
96
平均合併
57 分鐘
30 天內合併 PR
2

描述

## Background
I have fixed length of 200 bytes file with 100 multi segments present in copybook with 2k columns. It is taking almost 12 hours for just 1gb of file.
## Feature

Can you add feature for input file split in mb for all file formats, currently it is working for record format =VB, where input_split_size_mb

I tried to adjust block size in spark code, but cobrix taking default cluster block size.

## Proposed Solution [Optional]
Solution Ideas
1. Allow this input_split_size_mb for all file formats
2. How to take custom block size specified in spark configuration.
Ex:spark.conf.set(“dfs.blocksize”,”32m”)

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。