MaartenGr / MaartenGr/BERTopic

Show the distribution of top k documents in each topic

Open
#1,875 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Reference to https://github.com/MaartenGr/BERTopic/issues/93#:~:text=So%2C%20only%20set%20this%20to%20True%20if%20you%20have%20less%20than%20100.000%20documents., does it mean it will not compute probabilities when I have more than 100 documents (accurately, I have 40000+ documents) although setting calculate_probabilities=True?

My BERTopic model is:

umap_model = UMAP(n_neighbors=20, n_components=15, min_dist=0.0, metric='cosine', random_state=42)
cluster_model = KMeans(n_clusters=100, random_state=42)  # 100
vectorizer_model = CountVectorizer(stop_words="english")

ctfidf_model = ClassTfidfTransformer(
    seed_words=seed_words, 
    seed_multiplier=5
)

model = BERTopic(embedding_model='./paraphrase-MiniLM-L6-v2', 
                 umap_model=umap_model,
                 min_topic_size=50,
                 ctfidf_model=ctfidf_model,        
                 hdbscan_model=cluster_model,
                 vectorizer_model=vectorizer_model, 
                 calculate_probabilities=True
)

Then run this code:

topics, probabilities = model.fit_transform(abstracts)
print(probabilities)

The output is None.
Version:

# pip show bertopic
Name: bertopic
Version: 0.16.0
Summary: BERTopic performs topic Modeling with state-of-the-art transformer models.
Home-page: https://github.com/MaartenGr/BERTopic

I know how stressful it would be to compute all the probabilities. If the answer to the question at the beginning of this issue is yes, could you please add a parameter to let users set top k documents to show probabilities of most similar k documents for each topic?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at BERTopic's fit_transform entry point and reproduce the reported behavior with calculate_probabilities=True using the model configuration shown. Determine how a top-k probability result should be represented for 40,000+ documents, then verify that the requested output is produced without requiring the full probability matrix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.