MaartenGr / MaartenGr/BERTopic
reduce_outliers result has not been updated.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hello,
Because the quantity of -1s comprises approximately 50% of all the quantities, it is desired to reduce the outliers. The code is as follows:
from bertopic import BERTopic
import pandas as pd
import nltk
from nltk.corpus import stopwords
# 加载数据集
df1 = pd.read_csv("/content/drive/MyDrive/VR_origin_0115_remove_update_notime.csv")
text_column = 0
# 创建BERTopic模型
model = BERTopic(nr_topics="auto")
# 将文本转换为向量,并聚类
topics, probs = model.fit_transform(df1.iloc[:, text_column])
new_topics = model.reduce_outliers(df1.iloc[:, text_column],topics)
# 获取主题的关键词
model.get_topic_info()
After running the code, I found that the result remains the same as the original model and does not reflect the reduction of outliers. I would like to inquire about where the problem lies and how to resolve it.
Note:
1.The goal is to ensure that the final result obtained from get_topic_info reflects the reduction of outliers.
2.I have tried using topic_model.update_topics(docs, topics=new_topics), and while the result reflects the updated topics, the information corresponding to -1 is deleted. However, I still want to retain the outliers; I just wish to reduce the quantity of -1 occurrences due to their significant proportion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supplied BERTopic example and compare reduce_outliers, get_topic_info, and update_topics using the same topics and documents. Start by tracing how reduce_outliers updates topic assignments and how get_topic_info reads them; done means the reported -1 count is reduced while an outlier topic remains visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100