MaartenGr / MaartenGr/BERTopic
List index out of range in `find_topics` when called with a list of 1 string
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [x] I have searched and found no existing issues
### Desribe the bug
In general `find_topics` works, but when I call it with a list of just one string (e.g. `["computer science"]`), then it crashes.
Here is the code I use:
```
threshold = 0.65
ce_topics_df = pd.DataFrame()
for keywords_cat, keywords_words in keywords_cats.items():
logging.info(f"Finding topics for category: {keywords_cat}")
logging.info(f"Keywords: {keywords_words}")
similar_topics, similarities = topic_model.find_topics(keywords_words, top_n=30)
topdf = pd.DataFrame(
{
"Category": keywords_cat,
"Topic": similar_topics,
"Similarity": similarities,
}
).query("Similarity >= @threshold")
ce_topics_df = pd.concat([ce_topics_df, topdf], axis=0)
ce_topics_df = ce_topics_df.drop_duplicates(subset=["Category", "Topic"]).merge(topic_model.get_topic_info(), left_on="Topic", right_on="Topic", how="left")
```
and here is the log:
```
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[48], line 6
4 logging.info(f"Finding topics for category: {keywords_cat}")
5 logging.info(f"Keywords: {keywords_words}")
----> 6 similar_topics, similarities = topic_model.find_topics(keywords_words, top_n=30)
7 topdf = pd.DataFrame(
8 {
9 "Category": keywords_cat,
(...) 12 }
13 ).query("Similarity >= @threshold")
14 ce_topics_df = pd.concat([ce_topics_df, topdf], axis=0)
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\_bertopic.py:1467, in BERTopic.find_topics(self, search_term, image, top_n)
1465 # Extract search_term embeddings and compare with topic embeddings
1466 if search_term is not None:
-> 1467 search_embedding = self._extract_embeddings([search_term], method="word", verbose=False).flatten()
1468 elif image is not None:
1469 search_embedding = self._extract_embeddings(
1470 [None], images=[image], method="document", verbose=False
1471 ).flatten()
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\_bertopic.py:3709, in BERTopic._extract_embeddings(self, documents, images, method, verbose)
3707 embeddings = self.embedding_model.embed(documents=documents, images=images, verbose=verbose)
3708 elif method == "word":
-> 3709 embeddings = self.embedding_model.embed_words(words=documents, verbose=verbose)
3710 elif method == "document":
3711 embeddings = self.embedding_model.embed_documents(documents, verbose=verbose)
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\backend\_base.py:48, in BaseEmbedder.embed_words(self, words, verbose)
35 def embed_words(self, words: List[str], verbose: bool = False) -> np.ndarray:
36 """Embed a list of n words into an n-dimensional
37 matrix of embeddings.
38
(...) 46
47 """
---> 48 return self.embed(words, verbose)
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\backend\_sentencetransformers.py:84, in SentenceTransformerBackend.embed(self, documents, verbose)
72 def embed(self, documents: List[str], verbose: bool = False) -> np.ndarray:
73 """Embed a list of n documents/words into an n-dimensional
74 matrix of embeddings.
75
(...) 82 that each have an embeddings size of `m`
83 """
---> 84 embeddings = self.embedding_model.encode(documents, show_progress_bar=verbose)
85 return embeddings
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\torch\utils\_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs)
113 @functools.wraps(func)
114 def decorate_context(*args, **kwargs):
115 with ctx_factory():
--> 116 return func(*args, **kwargs)
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\SentenceTransformer.py:1020, in SentenceTransformer.encode(self, sentences, prompt_name, prompt, batch_size, show_progress_bar, output_value, precision, convert_to_numpy, convert_to_tensor, device, normalize_embeddings, truncate_dim, pool, chunk_size, **kwargs)
1018 for start_index in trange(0, len(sentences), batch_size, desc="Batches", disable=not show_progress_bar):
1019 sentences_batch = sentences_sorted[start_index : start_index + batch_size]
-> 1020 features = self.tokenize(sentences_batch, **kwargs)
1021 if self.device.type == "hpu":
1022 if "input_ids" in features:
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\SentenceTransformer.py:1570, in SentenceTransformer.tokenize(self, texts, **kwargs)
1559 """
1560 Tokenizes the texts.
1561
(...) 1567 "attention_mask", and "token_type_ids".
1568 """
1569 try:
-> 1570 return self[0].tokenize(texts, **kwargs)
1571 except TypeError:
1572 return self[0].tokenize(texts)
File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\models\Transformer.py:484, in Transformer.tokenize(self, texts, padding)
482 for text_tuple in texts:
483 batch1.append(text_tuple[0])
--> 484 batch2.append(text_tuple[1])
485 to_tokenize = [batch1, batch2]
487 # strip
IndexError: list index out of range
```
### Reproduction
```python
from bertopic import BERTopic
```
### BERTopic Version
0.17.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in bertopic/_bertopic.py at find_topics and follow the _extract_embeddings call into the embedding path shown in the traceback, including backend/_base.py and backend/_sentencetransformers.py. Reproduce the failure with a one-item string list and add a regression test; done means find_topics handles that input without IndexError and preserves its documented result shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100