MaartenGr / MaartenGr/BERTopic
OpenAI Representation: KeyError: 'content'
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I am getting a KeyError when running the topic model on a large dataset (400 k documents) (but not on smaller samples).
```
KeyError Traceback (most recent call last)
Cell In[11], line 1
----> 1 topic_labeller.fit(
2 data_train["content_truncated"]
3 )
File /mnt/batch/tasks/shared/LS_root/mounts/clusters/staudtc-dsh-search-n2/code/Users/Christian.Staudt.external/experimentation-search/cognitive-search/topic-labelling/cognitive_search_topic_labelling/model.py:222, in TopicLabeller.fit(self, docs, y)
219 if self.pre_trained_embeddings:
220 fit_args.update(dict(embeddings=self.pre_trained_embeddings))
--> 222 self.topic_model.fit(**fit_args)
224 return self
File /anaconda/envs/topic-labelling/lib/python3.10/site-packages/bertopic/_bertopic.py:303, in BERTopic.fit(self, documents, embeddings, images, y)
262 def fit(self,
263 documents: List[str],
264 embeddings: np.ndarray = None,
265 images: List[str] = None,
266 y: Union[List[int], np.ndarray] = None):
267 """ Fit the models (Bert, UMAP, and, HDBSCAN) on a collection of documents and generate topics
268
269 Arguments:
(...)
301 ```
302 """
--> 303 self.fit_transform(documents=documents, embeddings=embeddings, y=y, images=images)
304 return self
File /anaconda/envs/topic-labelling/lib/python3.10/site-packages/bertopic/_bertopic.py:411, in BERTopic.fit_transform(self, documents, embeddings, images, y)
408 self._save_representative_docs(custom_documents)
409 else:
410 # Extract topics by calculating c-TF-IDF
--> 411 self._extract_topics(documents, embeddings=embeddings)
413 # Reduce topics
414 if self.nr_topics:
File /anaconda/envs/topic-labelling/lib/python3.10/site-packages/bertopic/_bertopic.py:3296, in BERTopic._extract_topics(self, documents, embeddings, mappings)
3294 documents_per_topic = documents.groupby(['Topic'], as_index=False).agg({'Document': ' '.join})
3295 self.c_tf_idf_, words = self._c_tf_idf(documents_per_topic)
-> 3296 self.topic_representations_ = self._extract_words_per_topic(words, documents)
3297 self._create_topic_vectors(documents=documents, embeddings=embeddings, mappings=mappings)
3298 self.topic_labels_ = {key: f"{key}_" + "_".join([word[0] for word in values[:4]])
3299 for key, values in
3300 self.topic_representations_.items()}
File /anaconda/envs/topic-labelling/lib/python3.10/site-packages/bertopic/_bertopic.py:3586, in BERTopic._extract_words_per_topic(self, words, documents, c_tf_idf, calculate_aspects)
3584 self.topic_aspects_[aspect] = aspects
3585 elif isinstance(aspect_model, BaseRepresentation):
-> 3586 self.topic_aspects_[aspect] = aspect_model.extract_topics(self, documents, c_tf_idf, aspects)
3588 return topics
File /anaconda/envs/topic-labelling/lib/python3.10/site-packages/bertopic/representation/_openai.py:191, in OpenAI.extract_topics(self, topic_model, documents, c_tf_idf, topics)
189 else:
190 response = openai.ChatCompletion.create(**kwargs)
--> 191 label = response["choices"][0]["message"]["content"].strip().replace("topic: ", "")
192 else:
193 if self.exponential_backoff:
KeyError: 'content'
```
This is the representation model I am using:
```python
bertopic.representation.OpenAI(
model="gpt-35-turbo",
chat=True,
exponential_backoff=False,
# delay_in_seconds=1,
generator_kwargs={"engine": "gpt-35-turbo", "temperature": 0.1},
prompt=f"""
Output a concise, English topic label for the following keywords. Output only the label, format example for a label: Lorem Ipsum
Never make the label a list of keywords. Ensure short labels, single terms or short term combinations. Do not add a period at the end of the label. If you are unable to perform the task, output: None
[KEYWORDS]
""",
)
```
Contributor guide
Research direction
Start in bertopic/representation/_openai.py at the response parsing on line 191, then trace the ChatCompletion response produced by the configured OpenAI representation model. Reproduce the KeyError with the large dataset and determine what response shape occurs when the content field is absent; done means the failure is handled or reported clearly without breaking normal topic labeling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100