MaartenGr / MaartenGr/BERTopic

Persistent Zero Probability for Documents Previously Classified as Outliers after Implementing reduce_outliers and _map_probabilities

Open
#1,843 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Following the application of `topic_model.reduce_outliers` to minimize outliers and the subsequent updating of topic allocations probabilities with `_map_probabilities`, it is observed that documents which were initially classified as outliers (denoted by a topic of -1) and had a probability of zero, retain this zero probability post updating. This raises a question regarding whether this is an intended consequence of the operations, or if there may be a misunderstanding of the process involved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the implementations and documentation for topic_model.reduce_outliers and _map_probabilities, then trace how documents with topic -1 are handled across both operations. Reproduce the reported zero-probability behavior with a small outlier dataset and establish whether it matches the documented contract; completion requires a clear answer and, if incorrect, a regression test for the expected probabilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.