MaartenGr / MaartenGr/BERTopic

`hierarchical_topics()` parent nodes show raw c-TF-IDF keywords instead of representation model labels

Open
#2,503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Feature request

`hierarchical_topics()` builds parent topic names by concatenating the top 5 c-TF-IDF keywords — e.g., `"wear_safety_PPE_worker_work"`. Meanwhile, leaf topics get rich labels from the representation model (e.g., `"PPE Non-Compliance Incidents"` via an LLM). This creates a jarring inconsistency in `visualize_hierarchy()`.

Add a `use_representation_model` parameter that runs the representation pipeline on parent topics:

```python
# Current: parent nodes show "wear_safety_PPE_worker_work"
hierarchy = topic_model.hierarchical_topics(docs)

# Proposed: parent nodes get proper labels from the representation model
hierarchy = topic_model.hierarchical_topics(docs, use_representation_model=True)
```

### Motivation

**Root cause:** `_extract_words_per_topic` is called with `calculate_aspects=False` for parent nodes, so no aspect models (including LLM labeling) run on them.

When using LLM-based representation models, the hierarchy visualization becomes unusable because leaf labels (e.g., `"PPE Non-Compliance Incidents"`) are at a completely different abstraction level than parent labels (e.g., `"wear_safety_PPE_worker_work"`). Users must manually relabel parent nodes, which defeats the purpose of the representation pipeline.

### Your contribution

I can submit a PR that adds `use_representation_model: bool = False` to `hierarchical_topics()`. When enabled, a post-processing step runs the full representation pipeline on all parent topics in a **single batch call** — not inline during the merge loop (which would make N-1 separate LLM calls).

Default is `False` — existing behavior unchanged. The batch approach keeps LLM cost bounded.

**Note:** If the shared helpers refactoring (#2497) lands first, this PR can reuse `_aggregate_documents()`. Otherwise, the aggregation is inlined — either way works.

I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.

---

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with hierarchical_topics() and trace its _extract_words_per_topic(..., calculate_aspects=False) path for parent nodes. Review the proposed interaction with the representation pipeline and the shared helpers refactoring in #2497, while ensuring parent processing is batched rather than performed in the merge loop. Done means an opt-in use_representation_model parameter produces representation-model labels for parent nodes while preserving the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.