MaartenGr / MaartenGr/BERTopic
`reduce_outliers`: add target-based outlier reduction ("reduce to X% outliers")
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Feature request
`reduce_outliers()` requires manually specifying a `threshold` parameter. Users must run the method repeatedly, inspecting results each time, to find a threshold that achieves their desired outlier rate.
Add an `outliers_percentage_target` parameter that auto-searches for the optimal threshold:
```python
# Current: manual trial-and-error
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities", threshold=0.3)
# Check outlier rate... too high. Try 0.2... too low. Try 0.25...
# Proposed: specify the target directly
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities",
outliers_percentage_target=0.05) # "at most 5% outliers"
```
### Motivation
The trial-and-error workflow is evident in the 10+ open issues asking about `reduce_outliers` behavior:
- [#1785](https://github.com/MaartenGr/BERTopic/issues/1785) — `reduce_outliers` result not updated in model
- [#1843](https://github.com/MaartenGr/BERTopic/issues/1843) — persistent zero probability after `reduce_outliers`
- [#1520](https://github.com/MaartenGr/BERTopic/issues/1520) — clusters changed after reducing outliers
- [#2114](https://github.com/MaartenGr/BERTopic/issues/2114) — `reduce_outliers` removes stop_words/ngram_range effects
In production, users typically know their target — "I want at most 5% outliers" — not the internal threshold that achieves it.
### Your contribution
I can submit a PR that adds an `outliers_percentage_target` parameter (float, 0–1): auto-search for the threshold that achieves the target outlier percentage using binary search. Works with all 4 existing strategies (probabilities, distributions, c-tf-idf, embeddings) and requires no changes to the underlying reduction logic.
Cannot set both `threshold` and `outliers_percentage_target` — raises `ValueError`. Backward compatible: defaults to `None`, existing behavior unchanged.
I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.
---
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the implementation of reduce_outliers() and trace how threshold handling is shared across the probabilities, distributions, c-tf-idf, and embeddings strategies. Add the optional target behavior without changing existing calls; done means conflicting threshold and target inputs raise ValueError and each strategy can reach the requested outlier percentage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100