AbsaOSS / AbsaOSS/cobrix

Get checksum actual record bytes

Open
#749 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
170
Forks
96
Avg merge
57m
Merged PRs (30d)
2

Description

## Background
We have requirement like to validate upstream given checksum vs actual data checksum.

Currently we are stripping BDW and RDW and writing to file and calculating checksum.

## Feature

```Scala
def getDataBytes(infile:String,outfile:String) :String ={
val inputStream =new FileInputStream(new File(infile))
val outStream =new FileOutputStream(new File(outfile))
while (inputStream.available() > 0){
val bdw = BigInt(IOUtils.toByteArray(inputStream,2))
val bdwBuffer = Array.OfDim[Byte](bdw.toInt-2)
IOUtils.read(inputSream,bdwBuffer)
val block :Array[Byte] = bdwBuffer.drop(2)
var rdw =0
while (rdw < bdw -4) {
val rdwSize = BigInt(block.slice(rdw,rdw+2)).toInt
val record =block.slice(rdw+4, rdw+rdwSize)
outStream.write(record)
rdw = rdw + rdwSize
}
}
inputStream.close()
outStream.close()
}
```

Shell we avoid creating temp file and calculate directly record bytes hash (Md5 or Sha256) value ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.