apache / apache/hop

[Feature Request]: Add Checksum (Add same int hash method used in steps)

Open
#3,460 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 33m
Merged PRs (30d)
204

Description

### What would you like to happen?

I noticed all over the Hop code there are calls to java.lang.Object.hashCode() functions like this https://github.com/apache/hop/blob/b0711dcee31d4945e7458f25716323f2aae5a813/plugins/transforms/splunk/src/main/java/org/apache/hop/pipeline/transforms/splunkinput/ReturnValue.java#L64C31-L64C31

https://stackoverflow.com/questions/24922452/what-precisely-is-the-algorithm-used-by-java-lang-objects-hashcode

What I worry about is that no one knows what algorithm is being used. Certain transforms like the Combination Lookup / Update use a 'hashcode' stored as a 64bit int into the database. Not knowing leaves it a question of what is the collision space for these hashes. If I wanted to update a row and its hash outside of the transform, how would I do that?

So generally, the "Add Checksum" transform could receive new 'hash' function that is whatever is happening in those other transforms that can also output as an integer, not just a string or hex.

However, I think it a better solution to provide a dropdown in each of the transforms that use a hashcode to let people choose what algo they want. Preferably something like MD5 which is fast, has few enough collisions. I might add Sha-1 to that list.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Add Checksum transform and the Combination Lookup / Update transforms that store hashcode values, including the referenced Splunk ReturnValue.java example. Clarify whether the work should add an integer hash option, selectable algorithms in each relevant transform, or both. Done means the supported checksum behavior and algorithm choices are defined and implemented consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.