Process ASCII file with fixed length format
- Dominant language
- Scala
- Stars
- 170
- Forks
- 96
- Avg merge
- 57m
- Merged PRs (30d)
- 2
Description
## Background [Optional]
I have a need to use an ASCII file that has record segments of fixed length defined. There is no LF/CR kind of characters. The new record starts after every 426bytes. I believe, I should be able to ues Cobrix for this. I am looking for a documentation on what "options" should I specify in my spark.read method. I can see cobrix libraries working very well for ebcdic/binary cobol file. The files that I have are simple ASCII files. These files can grow upto 1GB easily. Spark parllel processsing will be very important for me. I do not have copybook for this defined. However, I know that the structure will be like bello2
Header -- size 100 bytes.
Records Segments -- each 426 bytes
Trailer - size 100 bytes (optional)
## Question
What spark.read options from cobrix can I use to process larger ASCII files with fixed or variable length record segments.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.