MaartenGr / MaartenGr/BERTopic

List index out of range in `find_topics` when called with a list of 1 string

Open
#2,392 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [x] I have searched and found no existing issues

### Desribe the bug

In general `find_topics` works, but when I call it with a list of just one string (e.g. `["computer science"]`), then it crashes.

Here is the code I use:

```
threshold = 0.65
ce_topics_df = pd.DataFrame()
for keywords_cat, keywords_words in keywords_cats.items():
logging.info(f"Finding topics for category: {keywords_cat}")
logging.info(f"Keywords: {keywords_words}")
similar_topics, similarities = topic_model.find_topics(keywords_words, top_n=30)
topdf = pd.DataFrame(
{
"Category": keywords_cat,
"Topic": similar_topics,
"Similarity": similarities,
}
).query("Similarity >= @threshold")
ce_topics_df = pd.concat([ce_topics_df, topdf], axis=0)
ce_topics_df = ce_topics_df.drop_duplicates(subset=["Category", "Topic"]).merge(topic_model.get_topic_info(), left_on="Topic", right_on="Topic", how="left")
```

and here is the log:

```
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[48], line 6
4 logging.info(f"Finding topics for category: {keywords_cat}")
5 logging.info(f"Keywords: {keywords_words}")
----> 6 similar_topics, similarities = topic_model.find_topics(keywords_words, top_n=30)
7 topdf = pd.DataFrame(
8 {
9 "Category": keywords_cat,
(...) 12 }
13 ).query("Similarity >= @threshold")
14 ce_topics_df = pd.concat([ce_topics_df, topdf], axis=0)

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\_bertopic.py:1467, in BERTopic.find_topics(self, search_term, image, top_n)
1465 # Extract search_term embeddings and compare with topic embeddings
1466 if search_term is not None:
-> 1467 search_embedding = self._extract_embeddings([search_term], method="word", verbose=False).flatten()
1468 elif image is not None:
1469 search_embedding = self._extract_embeddings(
1470 [None], images=[image], method="document", verbose=False
1471 ).flatten()

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\_bertopic.py:3709, in BERTopic._extract_embeddings(self, documents, images, method, verbose)
3707 embeddings = self.embedding_model.embed(documents=documents, images=images, verbose=verbose)
3708 elif method == "word":
-> 3709 embeddings = self.embedding_model.embed_words(words=documents, verbose=verbose)
3710 elif method == "document":
3711 embeddings = self.embedding_model.embed_documents(documents, verbose=verbose)

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\backend\_base.py:48, in BaseEmbedder.embed_words(self, words, verbose)
35 def embed_words(self, words: List[str], verbose: bool = False) -> np.ndarray:
36 """Embed a list of n words into an n-dimensional
37 matrix of embeddings.
38
(...) 46
47 """
---> 48 return self.embed(words, verbose)

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\bertopic\backend\_sentencetransformers.py:84, in SentenceTransformerBackend.embed(self, documents, verbose)
72 def embed(self, documents: List[str], verbose: bool = False) -> np.ndarray:
73 """Embed a list of n documents/words into an n-dimensional
74 matrix of embeddings.
75
(...) 82 that each have an embeddings size of `m`
83 """
---> 84 embeddings = self.embedding_model.encode(documents, show_progress_bar=verbose)
85 return embeddings

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\torch\utils\_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs)
113 @functools.wraps(func)
114 def decorate_context(*args, **kwargs):
115 with ctx_factory():
--> 116 return func(*args, **kwargs)

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\SentenceTransformer.py:1020, in SentenceTransformer.encode(self, sentences, prompt_name, prompt, batch_size, show_progress_bar, output_value, precision, convert_to_numpy, convert_to_tensor, device, normalize_embeddings, truncate_dim, pool, chunk_size, **kwargs)
1018 for start_index in trange(0, len(sentences), batch_size, desc="Batches", disable=not show_progress_bar):
1019 sentences_batch = sentences_sorted[start_index : start_index + batch_size]
-> 1020 features = self.tokenize(sentences_batch, **kwargs)
1021 if self.device.type == "hpu":
1022 if "input_ids" in features:

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\SentenceTransformer.py:1570, in SentenceTransformer.tokenize(self, texts, **kwargs)
1559 """
1560 Tokenizes the texts.
1561
(...) 1567 "attention_mask", and "token_type_ids".
1568 """
1569 try:
-> 1570 return self[0].tokenize(texts, **kwargs)
1571 except TypeError:
1572 return self[0].tokenize(texts)

File c:\Users\raffaele\venvs\bertopic\Lib\site-packages\sentence_transformers\models\Transformer.py:484, in Transformer.tokenize(self, texts, padding)
482 for text_tuple in texts:
483 batch1.append(text_tuple[0])
--> 484 batch2.append(text_tuple[1])
485 to_tokenize = [batch1, batch2]
487 # strip

IndexError: list index out of range
```

### Reproduction

```python
from bertopic import BERTopic

```

### BERTopic Version

0.17.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in bertopic/_bertopic.py at find_topics and follow the _extract_embeddings call into the embedding path shown in the traceback, including backend/_base.py and backend/_sentencetransformers.py. Reproduce the failure with a one-item string list and add a regression test; done means find_topics handles that input without IndexError and preserves its documented result shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.