MaartenGr / MaartenGr/BERTopic
partial_fit throws warning and inf scores
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [X] I have searched and found no existing issues
### Desribe the bug
running partial_fit starts to throw error after ~100 iterations
```
~/.venv/lib/python3.12/site-packages/bertopic/vectorizers/_ctfidf.py:84 RuntimeWarning: overflow encountered in divide
idf = np.log((avg_nr_samples / df) + 1)
```
### Reproduction
I'm on an M3 MacBook Pro
Python 3.12.4
scikit-learn 1.5.1
bertopic 0.16.3
numpy 1.26.4
scipy 1.14.0
- "df" is a pandas dataframe with a column "message", where each row is a string (in my use case, between 8 and 50 characters)
- "embeddings" is an np ndarray of shape [len(message), 384], of embeddings of the messages above. Embedded with the below embedding model:
```python
embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
```
here is a slightly modified version of the partial_fit example from the docs:
```python
from sklearn.cluster import MiniBatchKMeans
from sklearn.decomposition import IncrementalPCA
from bertopic.vectorizers import OnlineCountVectorizer
# Prepare sub-models that support online learning
umap_model = IncrementalPCA(n_components=5)
cluster_model = MiniBatchKMeans(n_clusters=50, random_state=0)
vectorizer_model = OnlineCountVectorizer(stop_words="english", decay=.01)
topic_model = BERTopic(umap_model=umap_model,
hdbscan_model=cluster_model,
vectorizer_model=vectorizer_model
)
batch_size = 128
for idx in range(0, len(df), batch_size):
docs = df["message"].iloc[idx:idx+batch_size]
embeds = embeddings[idx:idx+batch_size]
topic_model.partial_fit(list(docs.astype('string')), embeds)
```
### BERTopic Version
0.16.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with bertopic/vectorizers/_ctfidf.py at the reported line 84 and trace how partial_fit updates the online vectorizer. Run the supplied partial_fit example with the listed versions and inspect when the overflow and infinite scores begin. Done means the repeated updates no longer emit the warning and produce finite scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, numpy, pandas, python, scikit-learn
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100