`ValueError` during `partial_fit()` in `_create_topic_vectors`

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Desribe the bug

Summary

When calling BERTopic.partial_fit() on a new batch of documents and embeddings, the following error intermittently occurs during internal vector averaging:

ValueError: Shape of weights must be consistent with shape of a along specified axis.

Stack Trace (truncated)

  File "my_script.py", line 55, in <module>
    topic_model.partial_fit(documents=documents_B, embeddings=embeddings_B)
  File ".../bertopic/_bertopic.py", line 772, in partial_fit
    self._create_topic_vectors()
  File ".../bertopic/_bertopic.py", line 4214, in _create_topic_vectors
    topic_embedding = np.average(
        word_embeddings[i * n : n + (i * n)],
        weights=word_importance,
        axis=0,
    )
ValueError: Shape of weights must be consistent with shape of a along specified axis.

Description

I expected partial_fit() to successfully update the BERTopic model with new batches of documents and embeddings.

The method fails intermittently (in some iterations only) while inside a loop that performs partial_fit() multiple times with consistent data structures. The inputs (documents, embeddings) are non-empty, of matching lengths, and validated prior to the call. This suggests the bug is not a user-side data error but an internal mismatch during topic vector computation.

Hypothesized Cause

The issue appears to originate from self._create_topic_vectors(), where np.average(...) is called with weights=word_importance. This suggests that the number of topic words and the number of word embeddings do not match, possibly because:

• Some topic representation words generated during partial fitting cannot be embedded (e.g., due to being OOV, containing unusual tokens, or failing in the encoder)
• The internal method does not verify that all topic words were successfully embedded before averaging

Suggested Fix

Before computing the average with np.average(..., weights=...), ensure that the shape of word_embeddings matches the length of word_importance. If not, skip the topic, warn the user, or use uniform weighting with available vectors.

---

Environment

• bertopic version: 0.17.0
• Python version: 3.10
• OS: macOS

Reproduction

Pseudo code bellow:

from bertopic import BERTopic
from bertopic.representation import TextGeneration
from bertopic.representation import KeyBERTInspired
from bertopic.vectorizers import ClassTfidfTransformer
from bertopic.representation import PartOfSpeech
from bertopic.representation import MaximalMarginalRelevance
from sklearn.feature_extraction.text import CountVectorizer
from bertopic.vectorizers import OnlineCountVectorizer
from sklearn.cluster import MiniBatchKMeans
from sklearn.decomposition import IncrementalPCA

MIN_TOPIC_SIZE = 10
TOP_N_WORDS = 20
N_GRAMS_RANGE = (1, 2, 3)

KEY_BERT = KeyBERTInspired(top_n_words=TOP_N_WORDS)
CTFIDF_MODEL = ClassTfidfTransformer(reduce_frequent_words=False, bm25_weighting=True)

TOPIC_REPRESENTATION = {
    "Main": [KEY_BERT, MaximalMarginalRelevance(diversity=.5)],
    "c-tf-idf": KEY_BERT,
}

vectorizer_model = OnlineCountVectorizer(stop_words=STOP_WORDS, decay=.25)
dim_reduction_model = IncrementalPCA(n_components=15, batch_size=1000)
clustering_model = MiniBatchKMeans(n_clusters=8, random_state=42, batch_size=1000)

topic_model = BERTopic(
    embedding_model=EMBEDDING_MODEL,
    vectorizer_model=vectorizer_model,
    umap_model=dim_reduction_model,
    hdbscan_model=clustering_model,
    ctfidf_model=CTFIDF_MODEL,
    representation_model=TOPIC_REPRESENTATION,
    top_n_words=TOP_N_WORDS, 
    n_gram_range=N_GRAMS_RANGE, 
    min_topic_size=MIN_TOPIC_SIZE,
    calculate_probabilities=True,
    verbose=False
)

# Split dataframe into two dataframes at a certain point in time
period_A_df, period_B_df = split_period(period_df: pd.DataFrame, split_time: pd.Timestamp)

# "documents" column contains a pre-processed version of the "text" column
# "embeddings" columns has been computed with a custom OpenAI embedding class (that handles errors and token/calls) based on the column "documents"
documents_A = period_A["documents"].values.tolist()
embeddings_A = np.stack(period_A['embeddings'].values)
topic_model.fit(documents=documents_A, embeddings=embeddings_A)
# analyze stuffs here

documents_B = period_B["documents"].values.tolist()
embeddings_B = np.stack(period_B['embeddings'].values)
topic_model.partial_fit(documents=documents_B, embeddings=embeddings_B)

BERTopic Version

0.17.0

Start in bertopic/_bertopic.py at BERTopic.partial_fit() and _create_topic_vectors(), especially the np.average call around line 4214. Reproduce the intermittent failure with repeated partial_fit() calls using the supplied configuration, then verify that topic vector computation handles mismatched word embeddings and weights without raising the reported ValueError.