AbsaOSS / AbsaOSS/cobrix

EBCDIC Data Columns showing null for PIC 9(5) COMP-3

Open
#308 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
170
Forks
96
Avg merge
57m
Merged PRs (30d)
2

Description

**## Describe the bug**
Here are the Cobol copy book and ebcdic data files. I try to read using below statement. I was able to print the schema, however I have following 3 issues.

1. Not able to print CUST_ID , its showing as null. In the sample copyboock CUST_ID is PIC 9(5) COMP-3
2. Not able to parse other columns as per the copy book column fields length.
3. There are total 3 records in data file. but when we display its only showing single record.
However I used data stage I was able to parse and see the data as expected using same copy book and data.

Code:

val df = spark.read.format("cobol").option("schema_retention_policy", "collapse_root").option("is_record_sequence", true).option("copybook", "/myapp/new/sample_mainframe_fb_data_copybook.cob.txt").load("/myapp/new/sample_mainframe_fb_data.dat.txt")
[sample_mainframe_fb_data_copybook.cob.txt](https://github.com/AbsaOSS/cobrix/files/4899450/sample_mainframe_fb_data_copybook.cob.txt)
[sample_mainframe_fb_data.dat.txt](https://github.com/AbsaOSS/cobrix/files/4899460/sample_mainframe_fb_data.dat.txt)

## To Reproduce
Steps to reproduce the behaviour OR commands run:
1. Copy the Cobol and data files to your file system
2. Use the above code snippet in any of your running spark/scala program or spark-shell for interactive execution.

## Expected behaviour
A clear and concise description of what you expected to happen.
1. It should list all 3 records. 2. CUST_ID value able to readable, 3. View and align the other data columns as per the Cobol copybook

## Screenshots
If applicable, add screenshots to help explain your problem.

Results are showing like below

> ### +-------+------------+--------------------+------------------+--------+--------------------+------------+-----------+--------------------+
> |CUST_ID| CUST_NAME| STREET_ADDRESS| CITY|US_STATE|OTHER_STATE_PROVINCE|COUNTRY_CODE|POSTAL_CODE| NOTES|
> +-------+------------+--------------------+------------------+--------+--------------------+------------+-----------+--------------------+
> | null|[John, 1 Ma]|in St. Anyt|own NY| | USA1| 234| 5 This|is our first cust...|
> +-------+------------+--------------------+------------------+--------+--------------------+------------+-----------+--------------------+

## Additional context
Add any other context about the problem here.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.