corejavascript / corejavascript/typeahead.js

include uniqueness filter in noramlizeTokens

Open
#202 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
968
Forks
231
PR merge metrics
No merged PRs in 30d

Description

https://github.com/corejavascript/typeahead.js/blob/662ee462829fb977bdad59a5707db0db65335f9c/src/bloodhound/search_index.js#L136

I was looking at issue #183 regarding starts with logic. This seems to be a result of how the Ngram tokenizer works. I was playing around with implementing a tokenizer that would output interword tokens. It would generate a lot more tokens. I noticed normalizeTokens does not return unique list tokens. It seems like getIntersection efficiency would benefit from being fed a unique list? Would there be any downsides to this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.