TF-IDF: change to scikit-learn
- Dominant language
- Python
- Stars
- 134
- Forks
- 86
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 1
Description
Orange uses the following formula for IDF: `math.log10(number_of_docs/number_of_docs_with_word)`. In this case, some words become all 0 if they appear in all documents. This has them removed by subsequent preprocessors. To avoid this, one can use Smooth IDF, which uses `math.log10(1 + number_of_docs/number_of_docs_with_word)`.
Why is this a problem? This is not the same as in scikit.
a) IDF is `math.log10(number_of_docs/(number_of_docs_with_word + 1))`
b) Smooth is `math.log(1 + number_of_docs+1 / number_of_docs_with_word+1)`
c) Scikit uses natural log, while we use log10 (not a big issue, as all numbers are multiplied by constant, but still)
d) TF, when computing TF-IDF, is not normalized by document length, which is also a standard.
We should probably use scikit here. This would, of course, affect teaching materials.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the TF-IDF entry point and compare Orange's IDF, smoothing, logarithm, and TF normalization with scikit-learn. Review the teaching materials affected by the behavior; done means the implementation and materials consistently follow the agreed scikit-learn semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- documentation, machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100