MaartenGr / MaartenGr/BERTopic
`_extract_representative_docs` recomputes from scratch on every call — no caching
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Feature request
`_extract_representative_docs()` runs the full computation pipeline (sampling → c-TF-IDF → cosine similarity → MMR selection) every time it is invoked. In interactive workflows, users often call a sequence of visualization methods that each trigger this computation:
```python
topic_model.visualize_topics() # triggers representative docs computation
topic_model.visualize_hierarchy() # triggers it again
topic_model.visualize_documents() # triggers it again
```
Cache `representative_docs_` after first computation and invalidate when topic assignments change, following the same pattern already used for `topic_embeddings_`.
### Motivation
Each call repeats the same expensive similarity computations even though topic assignments haven't changed. This is also the root cause behind [#2367](https://github.com/MaartenGr/BERTopic/issues/2367) — KeyBERTInspired recomputes representative docs embeddings even when precomputed embeddings are available.
BERTopic already caches `topic_embeddings_` with compute-once-store-invalidate-on-change semantics. Representative docs should follow the same pattern.
### Your contribution
I can submit a PR that caches `representative_docs_` after first computation and invalidates the cache when topic assignments change (i.e., after `update_topics()`, `reduce_outliers()`, `merge_topics()`). Subsequent calls return the cached result without recomputation.
**Note:** This is most effective when the underlying `_extract_representative_docs` sampling and indexing bugs are fixed first (#2495). The caching mechanism itself is independent, but caching correct results is obviously preferable.
I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.
---
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating _extract_representative_docs() and the existing topic_embeddings_ compute-once-store-invalidate pattern. Trace update_topics(), reduce_outliers(), and merge_topics() to identify where topic assignments change, then verify repeated visualization calls reuse the cached result and those mutations invalidate it. The issue notes that a contributor is already prototyping the change and that #2495 affects the correctness of the cached result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100