BartMassey / BartMassey/simhash

Could you please clarify how the algorithm works?

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12
Forks
3
PR merge metrics
No merged PRs in 30d

Description

The manpage says:

> The algorithm used by simhash is Manassas' "shingleprinting" algorithm (see BIBLIOGRAPHY below): take a hash of every m-byte subsequence of the file, and retain the n of these hashes that are numerically smallest

Unfortunately, the url in the BIBLIOGRAPHY section (http://research.microsoft.com/research/sv/PageTurner/similarity.htm) is no longer reachable.

What I understand: You hash every subsequence of m bytes (with m=8 by default), and you keep a count of how many time each hash appears. Then you select n hashes (by default n=128) which will be your output. My question is: how do you select those n features?

What do you mean exactly when you say you retain the n "numerically smallest"? Do you select the n hashes that have appeared the least? Or do you select the n hashes that have the smallest value when interpreted as numbers?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the simhash manpage passage and its BIBLIOGRAPHY section, then compare the wording with the implementation if needed. Done means the documentation explicitly defines “numerically smallest,” explains how the n features are selected, and addresses the unavailable reference URL.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.