AbsaOSS / AbsaOSS/cobrix

Can I get the raw record bytes from ebcdic file w/out parsing

Đang mở
#656 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
question
Ngôn ngữ chính
Scala
Star
170
Fork
96
Merge trung bình
57 phút
Pull request đã merge (30 ngày)
2

Mô tả

I am trying to parse an ebcdic file for which I do not have a copybook. I know that whether it has RDW and/or BDW. It is one of the old legacy format file. We have written our own program that knows how to parse an individual record.

Is there a way that I can use Cobrix library only to parse and individual record in ebcdic bytes ? Once I get those bytes in rdd, I can write my 'map' function to parse the individual segments. I have used Cobrix library get an individual record. I have used following setup,

I have defined my copybook in a simple structute like below,

```
String copybook =
" 01 RECORD.\n" +
" 05 SEGMENT PIC X(1064).\n" ;

Dataset df1 = spark.read()
.format("za.co.absa.cobrix.spark.cobol.source")
.option("copybook_contents", copybook)
.option("encoding", "ebcdic")
.option("record_format", "V") // Variable length records
.option("is_rdw_big_endian", "true")
.option("rdw_adjustment", -4)
.option("schema_retention_policy", "collapse_root")
.load("data/Samples/sample_packed_variable_ebcdic_bigendian_rdw.dat");

```

I am able to parse the records based on the "RDW" value correctly. I get ROW object with only one element in it (as specified in my copybook with name SEGMENT). This SEGMENT is coming as string. I convert this string in "ibm500" character code set (convert to ebcdic) and parse it with the parsing program that we have written. Our program can parse the record based on the byte position. However, we are not able to parse the 'packed' decimal properly. It seems that the conversion from string to/from ebcdic bytes is losing the position and format. Is there a way for us to get origina raw bytes format as it appears in the file as part of the dataset that we get out. In shore, can 'SEGMENT' field in my example, represent the actual raw bytes for the entire record in the file ?

## Question
Is above method an acceptable way that I can use this library ? I like the Corbix's way of parallel processing of reading large ebcdic file and all I want help from this library is to parse rdw/bdw value and return the entire record in raw bytes that I can use for my own parsing logic. As I do not have proper copybood for the bytes segment.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.