MaartenGr / MaartenGr/BERTopic

Extract duplicated document-aggregation and topic-naming patterns into shared helpers

Open
#2,497 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Feature request

`_bertopic.py` contains several near-identical copy-paste patterns that have diverged slightly over time. Extracting them into shared helpers would reduce duplication and prevent future divergence.

Three patterns are affected:

1. **Document aggregation** — `documents.groupby(["Topic"], as_index=False).agg({"Document": " ".join})` appears in 4 methods: `_extract_topics`, `hierarchical_topics`, `update_topics`, and `partial_fit`. When a new column needs aggregating (e.g., images), every call site must be updated independently.

2. **Feature names extraction** — a 5-line sklearn version check (`get_feature_names` vs `get_feature_names_out`) is duplicated in `hierarchical_topics` and `_c_tf_idf`.

3. **Topic name from words** — `"_".join([x[0] for x in words][:N])` appears in 4 places across 2 methods (`topic_labels_` with `[:4]`, `hierarchical_topics` with `[:5]` in 3 places), with inconsistent slicing.

### Motivation

Reduce maintenance burden. Any change to the aggregation pattern (e.g., adding image support) currently requires updating 4 call sites in lock-step. The duplicated sklearn version check is a maintenance liability — when the old API is eventually dropped, two separate locations need updating. Inconsistent topic name slicing could produce subtly different results depending on the code path.

### Your contribution

I can submit a PR that extracts three private helpers:
- `_aggregate_documents(documents, columns=None)` — consolidates the groupby pattern
- `_get_feature_names(vectorizer)` — wraps the sklearn version check
- `_topic_name_from_words(words, n=5)` — consistent slicing

**Zero behavior change.** Each helper is a direct extraction of existing code. All existing tests pass without modification.

I've already been prototyping this in my fork, so I can turn it into a PR quickly if the direction looks good to you.

---

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in _bertopic.py with _extract_topics, hierarchical_topics, update_topics, partial_fit, and _c_tf_idf, comparing the duplicated aggregation, feature-name, and topic-name patterns. Extract the three proposed private helpers without behavior changes, then run the existing tests and confirm they still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.