MaartenGr / MaartenGr/BERTopic

Words not in CountVectorizer vocab despite being well above min_df threshold

Open
#1,665 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi Maarten,

I'm having an issue with some important words not appearing within CountVectorizer when using `min_df` even though they are well above the set threshold. My understanding of `min_df` is that when it is an integer (say, 10), all words that appear in at least 10 documents will be kept. Is that correct? See the example below,

```python
from umap import UMAP
from datasets import load_dataset
from bertopic import BERTopic
from sklearn.feature_extraction.text import CountVectorizer

dataset = load_dataset("CShorten/ML-ArXiv-Papers")["train"]
abstracts = dataset["abstract"][:5_000]

umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
vectorizer_model=CountVectorizer(min_df=10)

topic_model = BERTopic(umap_model=umap_model, vectorizer_model=vectorizer_model).fit(abstracts)
```

Take the word 'cancer' for example.

```python
# this under counts as it doesn't consider words next to punctuation,
# but wont consider words inside words (e.g. robot in robotics)
len([d for d in abstracts if " cancer " in d.lower()])
>>> 32
```

'cancer' appears in at least 32 documents (not counting occurrences up against punctuation so should be well above `min_df`.

```python
topic_model.vectorizer_model.vocabulary_["cancer"]
```
returns a key error.

Also try words: *privacy, reward, quantum, fuzzy, hashing, planning, player, robot, embeddings*

If I remove the min_df param and just use `CountVectorizer()`, then these words are all present.

I first came across this with the word 'rust' which was an important word in a large topic with ~70 document occurrences and `min_df=10`. It isn't prominent in the arxiv dataset though.

Do you know what is happening here?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided 5,000-abstract reproduction using BERTopic.fit, UMAP, and CountVectorizer(min_df=10). Compare the resulting vocabulary with a plain CountVectorizer and inspect how BERTopic processes the vectorizer output. Done means explaining the missing terms and confirming corrected min_df behavior without breaking the reported example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.