Process ASCII file with fixed length format
- Lenguaje dominante
- Scala
- Estrellas
- 170
- Forks
- 96
- Merge medio
- 57 min
- PR fusionados (30 d)
- 2
Descripción
## Background [Optional]
I have a need to use an ASCII file that has record segments of fixed length defined. There is no LF/CR kind of characters. The new record starts after every 426bytes. I believe, I should be able to ues Cobrix for this. I am looking for a documentation on what "options" should I specify in my spark.read method. I can see cobrix libraries working very well for ebcdic/binary cobol file. The files that I have are simple ASCII files. These files can grow upto 1GB easily. Spark parllel processsing will be very important for me. I do not have copybook for this defined. However, I know that the structure will be like bello2
Header -- size 100 bytes.
Records Segments -- each 426 bytes
Trailer - size 100 bytes (optional)
## Question
What spark.read options from cobrix can I use to process larger ASCII files with fixed or variable length record segments.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.