MaartenGr / MaartenGr/BERTopic

partial_fit throws warning and inf scores

Open
#2,098 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [X] I have searched and found no existing issues

### Desribe the bug

running partial_fit starts to throw error after ~100 iterations
```
~/.venv/lib/python3.12/site-packages/bertopic/vectorizers/_ctfidf.py:84 RuntimeWarning: overflow encountered in divide
idf = np.log((avg_nr_samples / df) + 1)
```

### Reproduction

I'm on an M3 MacBook Pro
Python 3.12.4
scikit-learn 1.5.1
bertopic 0.16.3
numpy 1.26.4
scipy 1.14.0

- "df" is a pandas dataframe with a column "message", where each row is a string (in my use case, between 8 and 50 characters)
- "embeddings" is an np ndarray of shape [len(message), 384], of embeddings of the messages above. Embedded with the below embedding model:
```python
embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
```

here is a slightly modified version of the partial_fit example from the docs:
```python
from sklearn.cluster import MiniBatchKMeans
from sklearn.decomposition import IncrementalPCA
from bertopic.vectorizers import OnlineCountVectorizer

# Prepare sub-models that support online learning
umap_model = IncrementalPCA(n_components=5)
cluster_model = MiniBatchKMeans(n_clusters=50, random_state=0)
vectorizer_model = OnlineCountVectorizer(stop_words="english", decay=.01)

topic_model = BERTopic(umap_model=umap_model,
hdbscan_model=cluster_model,
vectorizer_model=vectorizer_model
)

batch_size = 128
for idx in range(0, len(df), batch_size):
docs = df["message"].iloc[idx:idx+batch_size]
embeds = embeddings[idx:idx+batch_size]
topic_model.partial_fit(list(docs.astype('string')), embeds)
```

### BERTopic Version

0.16.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with bertopic/vectorizers/_ctfidf.py at the reported line 84 and trace how partial_fit updates the online vectorizer. Run the supplied partial_fit example with the listed versions and inspect when the overflow and infinite scores begin. Done means the repeated updates no longer emit the warning and produce finite scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, numpy, pandas, python, scikit-learn
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.