MaartenGr / MaartenGr/BERTopic
Zero topic distributions for some documents using approximate_distribution()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [X] I have searched and found no existing issues
### Desribe the bug
Dear creators of BERTopic,
Thanks for your work and this package is amazing. I have been using it for a long time. However, I found some documents (no matter whether they are used to train the model) have zero topic distributions for all topics created by BERTopic after applying `approximate_distribution()` function on them. It means that the topic distribution matrix produced by `approximate_distribution()` has some rows having sum of 0. Codes below did several things: (1) build a simple setup for a BERTopic model with PCA and KMEANS (from `cuML`) as the dimension reduction and clustering technique. (2) define a splitting function to split documents and pre-caculated embeddings. (3) fit the model on training data and compute topic distributions for both training and testing data set.
If more information is needed, please let me know. Thanks!
### Reproduction
```python
def pk(num_cluster):
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
umap_model = PCA(n_components = 10)
hdbscan_model = KMeans(n_clusters = num_cluster)
vectorizer_model = CountVectorizer()
Topic_model = BERTopic(embedding_model=embedding_model, umap_model=umap_model, hdbscan_model=hdbscan_model, vectorizer_model=vectorizer_model,
calculate_probabilities = False,verbose = True)
return Topic_model
def tr_te_split(documents,df,embeddings, i=1):
indices = np.arange(len(documents))
tr_ind, te_ind = train_test_split(indices, test_size=0.2, shuffle= True, random_state=i)
tr_df = df.iloc[tr_ind,:]
te_df = df.iloc[te_ind,:]
tr_documents = [documents[ind] for ind in tr_ind]
te_documents = [documents[ind] for ind in te_ind]
tr_embeddings = embeddings[tr_ind,:]
return tr_df,te_df,tr_documents,te_documents,tr_embeddings
def check_zero_exposure(arr):
if 0 in np.apply_along_axis(arr=arr,func1d=np.sum,axis=1):
return 1
zero_exposures = {}
for year in year_list:
df = pd.read_csv(df_folder+f"/contem_{year}_senti.csv")
documents = df.documents.tolist()
embeddings = np.load(embeddings_folder+f"/contem_{year}_senti_embeddings.npy")
tr_df, te_df, tr_documents,te_documents,tr_embeddings = tr_te_split(documents,df,embeddings)
tr_df.reset_index(drop=True,inplace=True)
te_df.reset_index(drop=True,inplace=True)
topic_model = pk(cluster_num)
topic_model.fit(tr_documents,tr_embeddings)
tr_topic_dist, _ = topic_model.approximate_distribution(tr_documents)
te_topic_dist, _ = topic_model.approximate_distribution(te_documents)
zero_exposure = [check_zero_exposure(tr_topic_dist),check_zero_exposure(te_topic_dist)]
zero_exposures[year] = zero_exposure
# zero_exposures
# {2014: [1, 1],
# 2015: [1, 1],
# 2016: [1, 1],
# 2017: [1, 1],
# 2018: [1, 1],
# 2019: [1, 1],
# 2020: [1, 1],
# 2021: [1, 1],
# 2022: [1, 1],
# 2023: [1, 1]}
```
### BERTopic Version
0.16.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named; start by locating approximate_distribution() and running the supplied BERTopic 0.16.2 reproduction with the PCA and KMeans setup. Trace how document-topic rows are produced and add a regression check for the reported zero-sum rows, with completion defined by the reproduction no longer producing them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100