AbsaOSS / AbsaOSS/cobrix

How can I load these multi-segment data from ASCII files?

オープン
#137 コメント 9 件 リアクション 0 件 担当者 0 名 GitHub で見る
question
主要言語
Scala
スター
170
フォーク
96
平均マージ
57分
マージ済み PR(30日)
2

説明

Hello,

Thank you for your work on this project, it is of great help for me.

So far, I have been able to successfully load ASCII single segment files, but failed with multi-segment ones.

Here is a simplified example of the kind of data I am trying to load into a DataFrame:

Copybook:
```
01 COMPANY-DETAILS.
05 SEGMENT-ID PIC 9(1).
05 STATIC-DETAILS.
10 NAME PIC X(2).

05 CONTACTS REDEFINES STATIC-DETAILS.
10 PERSON PIC X(3).
```

Data:
```
1BB
2CCC
```

Code:
```scala
val copybook =
""" 01 COMPANY-DETAILS.
| 05 SEGMENT-ID PIC 9(1).
| 05 STATIC-DETAILS.
| 10 NAME PIC X(2).
|
| 05 CONTACTS REDEFINES STATIC-DETAILS.
| 10 PERSON PIC X(3).
""".stripMargin

val df = spark.read
.format("cobol")
.option("copybook_contents", copybook)
.option("is_record_sequence", "true")
.option("schema_retention_policy", "collapse_root")
.option("encoding", "ascii")
.load("data_ascii/mini.txt")
```

Output:
```
+----------+--------------+--------+
|SEGMENT_ID|STATIC_DETAILS|CONTACTS|
+----------+--------------+--------+
| null| [2C]| [2CC]|
+----------+--------------+--------+
```
I can see 2 problems in my output:
* null value
* only one row : the 2 records seems to be read as if they were one (in my tests with my real data containing many records, I always end up with only one row in the dataframe)

After thoroughly reading your (very nice) README, I have tried to modify the copybook, data and several options, but I still fail to load my data correctly.

Since I am new to Cobol formats, I suspect either my use of Cobrix options to be incorrect, or my data format (ASCII, no record header in data) to be incompatible with Cobrix.

Can you see what is wrong here?

Thanks!

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。