[Feature Request]: Add Checksum (Add same int hash method used in steps)
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 33m
- Merged PRs (30d)
- 204
Description
### What would you like to happen?
I noticed all over the Hop code there are calls to java.lang.Object.hashCode() functions like this https://github.com/apache/hop/blob/b0711dcee31d4945e7458f25716323f2aae5a813/plugins/transforms/splunk/src/main/java/org/apache/hop/pipeline/transforms/splunkinput/ReturnValue.java#L64C31-L64C31
https://stackoverflow.com/questions/24922452/what-precisely-is-the-algorithm-used-by-java-lang-objects-hashcode
What I worry about is that no one knows what algorithm is being used. Certain transforms like the Combination Lookup / Update use a 'hashcode' stored as a 64bit int into the database. Not knowing leaves it a question of what is the collision space for these hashes. If I wanted to update a row and its hash outside of the transform, how would I do that?
So generally, the "Add Checksum" transform could receive new 'hash' function that is whatever is happening in those other transforms that can also output as an integer, not just a string or hex.
However, I think it a better solution to provide a dropdown in each of the transforms that use a hashcode to let people choose what algo they want. Preferably something like MD5 which is fast, has few enough collisions. I might add Sha-1 to that list.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start by reviewing the Add Checksum transform and the Combination Lookup / Update transforms that store hashcode values, including the referenced Splunk ReturnValue.java example. Clarify whether the work should add an integer hash option, selectable algorithms in each relevant transform, or both. Done means the supported checksum behavior and algorithm choices are defined and implemented consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100