MaartenGr / MaartenGr/BERTopic

Creating topics over time fails if model has deleted topics

Open
#2,530 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [x] I have searched and found no existing issues

### Desribe the bug

If the model has some of its topics deleted (via delete_topics), calling topics_over_time fails with error:
```
ValueError Traceback (most recent call last)
Cell In[37], line 1
----> 1 topics_over_time = topic_model.topics_over_time(tweets, timestamps, nr_bins=20)

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/bertopic/_bertopic.py:900, in BERTopic.topics_over_time(self, docs, timestamps, topics, nr_bins, datetime_format, evolution_tuning, global_tuning)
896 selection = documents.loc[documents.Timestamps == timestamp, :]
897 documents_per_topic = selection.groupby(["Topic"], as_index=False).agg(
898 {"Document": " ".join, "Timestamps": "count"}
899 )
--> 900 c_tf_idf, words = self._c_tf_idf(documents_per_topic, fit=False)
902 if global_tuning or evolution_tuning:
903 c_tf_idf = normalize(c_tf_idf, axis=1, norm="l1", copy=False)

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/bertopic/_bertopic.py:4451, in BERTopic._c_tf_idf(self, documents_per_topic, fit, partial_fit)
4448 if fit:
4449 self.ctfidf_model = self.ctfidf_model.fit(X, multiplier=multiplier)
-> 4451 c_tf_idf = self.ctfidf_model.transform(X)
4453 return c_tf_idf, words

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/sklearn/utils/_set_output.py:319, in _wrap_method_output..wrapped(self, X, *args, **kwargs)
317 @wraps(f)
318 def wrapped(self, X, *args, **kwargs):
--> 319 data_to_wrap = f(self, X, *args, **kwargs)
320 if isinstance(data_to_wrap, tuple):
321 # only wrap the first output for cross decomposition
322 return_tuple = (
323 _wrap_data_with_container(method, data_to_wrap[0], X, self),
324 *data_to_wrap[1:],
325 )

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/bertopic/vectorizers/_ctfidf.py:113, in ClassTfidfTransformer.transform(self, X)
110 if self.reduce_frequent_words:
111 X.data = np.sqrt(X.data)
--> 113 X = X * self._idf_diag
115 return X

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/scipy/sparse/_matrix.py:55, in spmatrix.__mul__(self, other)
54 def __mul__(self, other):
---> 55 return self._matmul_dispatch(other)

File ~/miniconda3/envs/tgram/lib/python3.13/site-packages/scipy/sparse/_base.py:911, in _spbase._matmul_dispatch(self, other)
909 if issparse(other):
910 if N != other.shape[0]:
--> 911 raise ValueError(
912 f"{err_prefix} (n,k={N}),(k={other.shape[0]},m)->(n,m)"
913 )
914 return self._matmul_sparse(other)
916 # If it's a list or whatever, treat it like an array

ValueError: matmul: dimension mismatch with signature (n,k=30856),(k=30855,m)->(n,m)
```

### Reproduction

```python
from bertopic import BERTopic
import re
import pandas as pd

# Prepare data
trump = pd.read_csv('https://drive.google.com/uc?export=download&id=1xRKHaP-QwACMydlDnyFPEaFdtskJuBa6')
trump.text = trump.apply(lambda row: re.sub(r"http\S+", "", row.text).lower(), 1)
trump.text = trump.apply(lambda row: " ".join(filter(lambda x:x[0]!="@", row.text.split())), 1)
trump.text = trump.apply(lambda row: " ".join(re.sub("[^a-zA-Z]+", " ", row.text).split()), 1)
trump = trump.loc[(trump.isRetweet == "f") & (trump.text != ""), :]
timestamps = trump.date.to_list()
tweets = trump.text.to_list()

topic_model = BERTopic(verbose=True)
topics, probs = topic_model.fit_transform(tweets)

topic_model.delete_topics([7])

topics_over_time = topic_model.topics_over_time(tweets, timestamps, nr_bins=20) # throws the error
topic_model.visualize_topics_over_time(topics_over_time, top_n_topics=20)

```

### BERTopic Version

0.17.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied example with BERTopic 0.17.4, then inspect the topics_over_time entry point in bertopic/_bertopic.py and the transform path in bertopic/vectorizers/_ctfidf.py. Trace the topic and c-TF-IDF dimensions after delete_topics([7]); done means topics_over_time completes and the subsequent visualization call works without the dimension-mismatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.