AbsaOSS / AbsaOSS/cobrix

Can the writer support the scenario when the output is a positional file with header&data?

未關閉
#843 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
question
主要語言
Scala
星號
170
分支
96
平均合併
57 分鐘
30 天內合併 PR
2

描述

## Question
Can the writer support the scenario when the output is a positional file with header&data?

Considering header and data as two different sets of records having different schemas, usually header is 1 row while data is N row.

What I was considering:
- manage header and data in same Spark Dataframe. Resulting in a Df with schema the union of columns of header and columns of data. Headers column will have values only in first row and nulls in the rest N rows, while data columns will have nulls in first row and values only in N rows after the first one (where N is the number of records in my df)
- write the df with 2 different copybooks: 1 with the header schema (writing only top 1 row), 1 with the data schema (writing all rows except first one). when writing all the columns not in the copybook will not be written to outputs
- the results: 2 separate files
- merge them together following the order header>data.

Do you think it will work? Is there something smarter?

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。