MaartenGr / MaartenGr/BERTopic

reduce_outliers result has not been updated.

Open
#1,785 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hello,
Because the quantity of -1s comprises approximately 50% of all the quantities, it is desired to reduce the outliers. The code is as follows:

from bertopic import BERTopic
import pandas as pd
import nltk
from nltk.corpus import stopwords

# 加载数据集
df1 = pd.read_csv("/content/drive/MyDrive/VR_origin_0115_remove_update_notime.csv")
text_column = 0
# 创建BERTopic模型
model = BERTopic(nr_topics="auto")
# 将文本转换为向量,并聚类
topics, probs = model.fit_transform(df1.iloc[:, text_column])
new_topics = model.reduce_outliers(df1.iloc[:, text_column],topics)
# 获取主题的关键词

model.get_topic_info()

After running the code, I found that the result remains the same as the original model and does not reflect the reduction of outliers. I would like to inquire about where the problem lies and how to resolve it.

Note:
1.The goal is to ensure that the final result obtained from get_topic_info reflects the reduction of outliers.
2.I have tried using topic_model.update_topics(docs, topics=new_topics), and while the result reflects the updated topics, the information corresponding to -1 is deleted. However, I still want to retain the outliers; I just wish to reduce the quantity of -1 occurrences due to their significant proportion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supplied BERTopic example and compare reduce_outliers, get_topic_info, and update_topics using the same topics and documents. Start by tracing how reduce_outliers updates topic assignments and how get_topic_info reads them; done means the reported -1 count is reduced while an outlier topic remains visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.