MaartenGr / MaartenGr/BERTopic
`hierarchical_topics()` parent nodes show raw c-TF-IDF keywords instead of representation model labels
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Feature request
`hierarchical_topics()` builds parent topic names by concatenating the top 5 c-TF-IDF keywords — e.g., `"wear_safety_PPE_worker_work"`. Meanwhile, leaf topics get rich labels from the representation model (e.g., `"PPE Non-Compliance Incidents"` via an LLM). This creates a jarring inconsistency in `visualize_hierarchy()`.
Add a `use_representation_model` parameter that runs the representation pipeline on parent topics:
```python
# Current: parent nodes show "wear_safety_PPE_worker_work"
hierarchy = topic_model.hierarchical_topics(docs)
# Proposed: parent nodes get proper labels from the representation model
hierarchy = topic_model.hierarchical_topics(docs, use_representation_model=True)
```
### Motivation
**Root cause:** `_extract_words_per_topic` is called with `calculate_aspects=False` for parent nodes, so no aspect models (including LLM labeling) run on them.
When using LLM-based representation models, the hierarchy visualization becomes unusable because leaf labels (e.g., `"PPE Non-Compliance Incidents"`) are at a completely different abstraction level than parent labels (e.g., `"wear_safety_PPE_worker_work"`). Users must manually relabel parent nodes, which defeats the purpose of the representation pipeline.
### Your contribution
I can submit a PR that adds `use_representation_model: bool = False` to `hierarchical_topics()`. When enabled, a post-processing step runs the full representation pipeline on all parent topics in a **single batch call** — not inline during the merge loop (which would make N-1 separate LLM calls).
Default is `False` — existing behavior unchanged. The batch approach keeps LLM cost bounded.
**Note:** If the shared helpers refactoring (#2497) lands first, this PR can reuse `_aggregate_documents()`. Otherwise, the aggregation is inlined — either way works.
I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.
---
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with hierarchical_topics() and trace its _extract_words_per_topic(..., calculate_aspects=False) path for parent nodes. Review the proposed interaction with the representation pipeline and the shared helpers refactoring in #2497, while ensuring parent processing is batched rather than performed in the merge loop. Done means an opt-in use_representation_model parameter produces representation-model labels for parent nodes while preserving the default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100