MaartenGr / MaartenGr/BERTopic
Show the distribution of top k documents in each topic
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Reference to https://github.com/MaartenGr/BERTopic/issues/93#:~:text=So%2C%20only%20set%20this%20to%20True%20if%20you%20have%20less%20than%20100.000%20documents., does it mean it will not compute probabilities when I have more than 100 documents (accurately, I have 40000+ documents) although setting calculate_probabilities=True?
My BERTopic model is:
umap_model = UMAP(n_neighbors=20, n_components=15, min_dist=0.0, metric='cosine', random_state=42)
cluster_model = KMeans(n_clusters=100, random_state=42) # 100
vectorizer_model = CountVectorizer(stop_words="english")
ctfidf_model = ClassTfidfTransformer(
seed_words=seed_words,
seed_multiplier=5
)
model = BERTopic(embedding_model='./paraphrase-MiniLM-L6-v2',
umap_model=umap_model,
min_topic_size=50,
ctfidf_model=ctfidf_model,
hdbscan_model=cluster_model,
vectorizer_model=vectorizer_model,
calculate_probabilities=True
)
Then run this code:
topics, probabilities = model.fit_transform(abstracts)
print(probabilities)
The output is None.
Version:
# pip show bertopic
Name: bertopic
Version: 0.16.0
Summary: BERTopic performs topic Modeling with state-of-the-art transformer models.
Home-page: https://github.com/MaartenGr/BERTopic
I know how stressful it would be to compute all the probabilities. If the answer to the question at the beginning of this issue is yes, could you please add a parameter to let users set top k documents to show probabilities of most similar k documents for each topic?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at BERTopic's fit_transform entry point and reproduce the reported behavior with calculate_probabilities=True using the model configuration shown. Determine how a top-k probability result should be represented for 40,000+ documents, then verify that the requested output is produced without requiring the full probability matrix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100