MaartenGr / MaartenGr/BERTopic

Add top-down (divisive) strategy for `hierarchical_topics()`

Open
#2,508 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Feature request

hierarchical_topics() builds hierarchies exclusively bottom-up using scipy's agglomerative linkage on c-TF-IDF vectors. Add a top-down (divisive) alternative:

# Current behavior (default)
hierarchy = topic_model.hierarchical_topics(docs, strategy="agglomerative")

# New: top-down recursive splitting
hierarchy = topic_model.hierarchical_topics(docs, strategy="divisive")
Motivation

Agglomerative linkage has known limitations:

  • Merge quality degrades at higher levels — late merges combine dissimilar topics because linkage minimizes global distance, not local semantic coherence
  • No per-split representation — parent nodes get concatenated keywords
  • #1907 — confirmed bug when 3+ topics have identical c-TF-IDF distances, breaking the hierarchy

A top-down approach avoids these issues by recursively splitting topics where each split is locally optimal.

Your contribution

I can submit a PR that adds a strategy parameter to hierarchical_topics(). The divisive path recursively splits topics using c-TF-IDF weighted NMF decomposition, building a tree where each parent-child relationship reflects a meaningful topic subdivision.

Default is "agglomerative" — existing behavior unchanged.

I've already been prototyping this in my fork (working implementation with tests). Since the divisive path is a new algorithm to maintain, I'm happy to discuss scope (e.g. landing it as experimental first) before opening the PR.


Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the hierarchical_topics() entry point and read how the current agglomerative strategy builds the hierarchy from c-TF-IDF vectors. Review the proposed divisive path and its existing prototype tests. Done means supporting strategy="divisive" while preserving agglomerative as the default and covering the new behavior with tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.