AbsaOSS / AbsaOSS/cobrix

variable length copybook

Open
#141 15 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
170
Forks
96
Avg merge
57m
Merged PRs (30d)
2

Description

When I tried to parse variable length file with rdw headers. I tried like below

cobolDataframe = spark
.read
.format("cobol")
.option("copybook", v_copybook)
.option("schema_retention_policy", "collapse_root") //removes the root record headerc
.option("drop_group_fillers", "false")
.option("generate_record_id", false) // this adds the file id and record id
.option("is_record_sequence", "true") // reader to use 4 byte record headers to extract records from a mainframe file
.option("is_rdw_big_endian", "true")
.option("is_rdw_part_of_record_length", true)
.option("rdw_adjustment", -4)
.load(v_data)

After parsing the data is not correct. I tried by having different rdw options but in all case I did not get correct parsed data.
Same file if I use with recordlength option
.option("record_length_field",v_recordLengthField)
its parsing correctly. But all of our mainframe files does not have record length field. How can I resolve this. so variable length files should work.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.