MaartenGr / MaartenGr/BERTopic

`approximate_distribution`: add target-based min_similarity ("at most N unassigned documents")

Open
#2,502 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Feature request

`approximate_distribution()` requires manually specifying `min_similarity`. Like the `threshold` parameter in `reduce_outliers`, users must iterate to find a value that produces an acceptable number of documents with zero topic distribution.

Add an `outliers_nb_target` parameter that auto-searches for the optimal `min_similarity`:

```python
# Current: manual trial-and-error
distributions = topic_model.approximate_distribution(docs, min_similarity=0.1)
# Count zeros... too many. Try 0.05... too few. Try 0.08...

# Proposed: specify the target directly
distributions = topic_model.approximate_distribution(docs, outliers_nb_target=50)
```

### Motivation

Users know their target — "I want at most 50 documents without any topic assignment" — not the internal `min_similarity` value that achieves it. The same trial-and-error UX problem as `reduce_outliers` (see related issue).

### Your contribution

I can submit a PR that adds:
- `outliers_nb_target` (int): target number of zero-distribution documents
- **Performance optimization:** refactor the internals so that the expensive similarity matrix is computed once and reused across all binary search iterations

Cannot set both `min_similarity` and `outliers_nb_target`. Backward compatible: defaults to `None`.

I've already been prototyping this in my fork, so I can open a PR quickly if this looks like a good direction.

---

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the approximate_distribution() implementation and compare its threshold-search behavior with reduce_outliers. Define how outliers_nb_target interacts with min_similarity, then verify that the similarity matrix is reused across the search and that the default None behavior remains backward compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.