MaartenGr / MaartenGr/BERTopic
Words not in CountVectorizer vocab despite being well above min_df threshold
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi Maarten,
I'm having an issue with some important words not appearing within CountVectorizer when using `min_df` even though they are well above the set threshold. My understanding of `min_df` is that when it is an integer (say, 10), all words that appear in at least 10 documents will be kept. Is that correct? See the example below,
```python
from umap import UMAP
from datasets import load_dataset
from bertopic import BERTopic
from sklearn.feature_extraction.text import CountVectorizer
dataset = load_dataset("CShorten/ML-ArXiv-Papers")["train"]
abstracts = dataset["abstract"][:5_000]
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
vectorizer_model=CountVectorizer(min_df=10)
topic_model = BERTopic(umap_model=umap_model, vectorizer_model=vectorizer_model).fit(abstracts)
```
Take the word 'cancer' for example.
```python
# this under counts as it doesn't consider words next to punctuation,
# but wont consider words inside words (e.g. robot in robotics)
len([d for d in abstracts if " cancer " in d.lower()])
>>> 32
```
'cancer' appears in at least 32 documents (not counting occurrences up against punctuation so should be well above `min_df`.
```python
topic_model.vectorizer_model.vocabulary_["cancer"]
```
returns a key error.
Also try words: *privacy, reward, quantum, fuzzy, hashing, planning, player, robot, embeddings*
If I remove the min_df param and just use `CountVectorizer()`, then these words are all present.
I first came across this with the word 'rust' which was an important word in a large topic with ~70 document occurrences and `min_df=10`. It isn't prominent in the arxiv dataset though.
Do you know what is happening here?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided 5,000-abstract reproduction using BERTopic.fit, UMAP, and CountVectorizer(min_df=10). Compare the resulting vocabulary with a plain CountVectorizer and inspect how BERTopic processes the vectorizer output. Done means explaining the missing terms and confirming corrected min_df behavior without breaking the reported example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100