MaartenGr / MaartenGr/BERTopic
Add top-down (divisive) strategy for `hierarchical_topics()`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Feature request
hierarchical_topics() builds hierarchies exclusively bottom-up using scipy's agglomerative linkage on c-TF-IDF vectors. Add a top-down (divisive) alternative:
# Current behavior (default)
hierarchy = topic_model.hierarchical_topics(docs, strategy="agglomerative")
# New: top-down recursive splitting
hierarchy = topic_model.hierarchical_topics(docs, strategy="divisive")
Motivation
Agglomerative linkage has known limitations:
- Merge quality degrades at higher levels — late merges combine dissimilar topics because linkage minimizes global distance, not local semantic coherence
- No per-split representation — parent nodes get concatenated keywords
- #1907 — confirmed bug when 3+ topics have identical c-TF-IDF distances, breaking the hierarchy
A top-down approach avoids these issues by recursively splitting topics where each split is locally optimal.
Your contribution
I can submit a PR that adds a strategy parameter to hierarchical_topics(). The divisive path recursively splits topics using c-TF-IDF weighted NMF decomposition, building a tree where each parent-child relationship reflects a meaningful topic subdivision.
Default is "agglomerative" — existing behavior unchanged.
I've already been prototyping this in my fork (working implementation with tests). Since the divisive path is a new algorithm to maintain, I'm happy to discuss scope (e.g. landing it as experimental first) before opening the PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the hierarchical_topics() entry point and read how the current agglomerative strategy builds the hierarchy from c-TF-IDF vectors. Review the proposed divisive path and its existing prototype tests. Done means supporting strategy="divisive" while preserving agglomerative as the default and covering the new behavior with tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100