dotnet / dotnet/machinelearning

Hash a vector to a scalar

Open
#1,322 4 comments 0 reactions 0 assignees View on GitHub
area-Transforms enhancement Priority:3
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

There is an existing issue #1031 that mentions that hashing a vector input to a scalar as a desirable output. While we can expand the set of types supported by hashing immediately (as done in #1303), the problem of how to hash vectors to a scalar is surprisingly thorny.

The fundamental root of all difficulties lies in the idea that a sparse vector must be considered logically equivalent to a dense vector of the same length. So: the length 4 sparse vector `{1:"hello", 3:"friend"}` should hash to the same value as the dense vector `{"", "hello", "", "friend"}`. Yet it should not hash to the same value as `{"hello", "", "", "friend"}`, or `{"", "hello", "", "friend", ""}`, or some other such inputs.

Beyond handling of how to handle sparse vectors and dense vectors with default values, there is also the problem of how to handle the "not ordered" option. The primary reason the not ordered option is valuable is because we want to use the hash transform in place of a dictionary-based term transform, to enable a bag of words representation. Yet if the ability to map both `{"a", "b", "a"}` and `{"a", "a", "b"}` to the same output vector (in the case of bag-of-words) is valuable, would mapping both to the same hash also likewise be a valuable property? If we *don't* believe this is valuable, what does being unordered in the case of a single hash actually mean? Nothing? Should it be disallowed, ignored?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the related issues #1031 and #1303 and the existing hash transform behavior. Done requires resolving sparse-versus-dense equivalence and defining the meaning of the unordered option before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.