openAI BERTopic coherence score

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Desribe the bug

Hi. I am trying to perform BERTopic using gpt model represenattion. I got the topics, but my issue is the coherence. I cannot underestand what is the issue. please guide me.

ValueError Traceback (most recent call last)
Cell In[20], line 33
30 topic_words_ids = [[dictionary.token2id[word] for word in topic if word in dictionary.token2id] for topic in topic_words]
32 # Calculate the coherence score using Gensim's CoherenceModel
---> 33 coherence_model = CoherenceModel(topics=topic_words_ids, texts=texts, dictionary=dictionary, coherence='c_v')
34 coherence_score = coherence_model.get_coherence()
36 print(f"Coherence Score: {coherence_score}")

File ~\AppData\Roaming\Python\Python310\site-packages\gensim\models\coherencemodel.py:214, in CoherenceModel.init(self, model, topics, texts, corpus, dictionary, window_size, keyed_vectors, coherence, topn, processes)
212 self._accumulator = None
213 self._topics = None
--> 214 self.topics = topics
216 self.processes = processes if processes >= 1 else max(1, mp.cpu_count() - 1)

File ~\AppData\Roaming\Python\Python310\site-packages\gensim\models\coherencemodel.py:429, in CoherenceModel.topics(self, topics)
427 new_topics = []
428 for topic in topics:
--> 429 topic_token_ids = self._ensure_elements_are_ids(topic)
430 new_topics.append(topic_token_ids)
432 if self.model is not None:

File ~\AppData\Roaming\Python\Python310\site-packages\gensim\models\coherencemodel.py:453, in CoherenceModel._ensure_elements_are_ids(self, topic)
451 return np.array(ids_from_ids)
452 else:
--> 453 raise ValueError('unable to interpret topic as either a list of tokens or a list of ids')

ValueError: unable to interpret topic as either a list of tokens or a list of ids

### BERTopic Version

0.16.3

Start with the reported notebook traceback at the Gensim CoherenceModel call and review how topic_words_ids is assembled from topic_words and dictionary.token2id. Reproduce the error with BERTopic 0.16.3, then determine whether the failure is caused by the reported input conversion or by BERTopic; done means a confirmed cause and a documented resolution or maintainer guidance.