Allow Neighbors to accept sparse data
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1.1k
- Avg merge
- 12d 2h
- Merged PRs (30d)
- 1
Description
**What's your use case?**
I want to use Neighbors to search a corpus of documents for items similar to one or more reference documents. Since Neighbors requires that Reference and Data have the same features, I have to apply either Text Embedding, Similarity Hashing or Topic Modeling in order to represent the corpora quantitatively. But for most ML tasks with text, I find Bag of Words usually producing more convincing results.
**What's your proposed solution?**
Allow Neighbors to accept datasets with different features, at least when it comes to sparse data from Bag of Words. So, before computing distances, the words that are in Reference but not in Data are added to Data with value 0, and the other way around.
**Are there any alternative solutions?**
Not that I'm aware of.
Contributor guide
Research direction
Start by locating the Neighbors widget and reading how it currently handles Reference and Data feature compatibility. Compare that behavior with the proposed Bag-of-Words use case and determine how sparse features would be aligned before distance computation. Done means Neighbors accepts compatible sparse datasets with differing feature sets and preserves the existing behavior for matching features.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100