MaartenGr / MaartenGr/BERTopic
ctfidf breaks down when specifying a vocabulary in CountVectorizer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
In some cases, the stop_words parameter of the CountVectorizer is not enough to prevent certain non-desired words from coming through. For example, one may have the desire to filter out non-verbs like abbreviations before coming up with topic representations.
This can be done by specifying a vocubulary in the CountVectorizer object
[sklearn docs](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)
However, a problem that occurs then is that ctfidf breaks down due to division by zero in line 82 of _ctfidf.py:
` idf = np.log((avg_nr_samples / df)+1)`
because it could be that some words in the vocabulary actually never occur.
I would therefore propose to change the line above to
` idf = np.log((avg_nr_samples / np.maximum(df, 1))+1)`
This solution does not change behaviour in normal cases and gives the optionality to specify a vocabulary when creating topic representations
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in _ctfidf.py at line 82 and inspect how df is produced when CountVectorizer receives a vocabulary containing unused words. Confirm the division-by-zero failure, then verify that the c-TF-IDF calculation remains unchanged for normally occurring vocabulary terms and handles unused terms without failing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100