MaartenGr / MaartenGr/BERTopic

Better error message when passing a df to docs

Open
#1,589 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Many methods like `visualize_documents` take a `docs` argument which should be a list, but most of the time my documents are stored in a `pd.dataframe` because there is other metadata associated with them and I often inadvertently end up passing the data frame to these methods rather than the list. Even though there is type checking on the input:

https://github.com/MaartenGr/BERTopic/blob/62e97ddea6cdcf9e4da25f9eaed478b22a9f9e20/bertopic/plotting/_documents.py#L9C1-L11C50

It doesn't throw a helpful error (e.g. 'docs should be type List'), instead it throws a KeyError with a random number, which often takes an embarrassing amount of time trying to debug before I remember that I've been here before and know what I've done wrong.

Is there any chance that all of these methods could either do some stricter type checking or check for a dataframe on that input?

```python
from sklearn.datasets import fetch_20newsgroups
import pandas as pd
from bertopic import BERTopic

data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))
docs = data['data'][0:500]
docs_df = pd.DataFrame({"docs": docs, "year": 2000})

topic_model = BERTopic()
topics, _ = topic_model.fit_transform(docs)

topic_model.visualize_documents(docs_df)
```

```
KeyError Traceback (most recent call last)
File [c:\Users\abb064\AppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\pandas\core\indexes\base.py:3790](file:///C:/Users/abb064/AppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/pandas/core/indexes/base.py:3790), in Index.get_loc(self, key)
3789 try:
-> 3790 return self._engine.get_loc(casted_key)
3791 except KeyError as err:

File index.pyx:152, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:181, in pandas._libs.index.IndexEngine.get_loc()

File pandas\_libs\hashtable_class_helper.pxi:7080, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\_libs\hashtable_class_helper.pxi:7088, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 413

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)

File [c:\Users\XXXAppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\bertopic\_bertopic.py:2286](file:///C:/Users/XXXAppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/bertopic/_bertopic.py:2286), in BERTopic.visualize_documents(self, docs, topics, embeddings, reduced_embeddings, sample, hide_annotations, hide_document_hover, custom_labels, title, width, height)
2216 """ Visualize documents and their topics in 2D
2217
2218 Arguments:
(...)
2283 style="width:1000px; height: 800px; border: 0px;"">
2284 """
2285 check_is_fitted(self)
-> 2286 return plotting.visualize_documents(self,
2287 docs=docs,
2288 topics=topics,
2289 embeddings=embeddings,
2290 reduced_embeddings=reduced_embeddings,
2291 sample=sample,
2292 hide_annotations=hide_annotations,
2293 hide_document_hover=hide_document_hover,
2294 custom_labels=custom_labels,
2295 title=title,
2296 width=width,
2297 height=height)

File [c:\Users\XXXAppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\bertopic\plotting\_documents.py:105](file:///C:/Users/XXXAppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/bertopic/plotting/_documents.py:105), in visualize_documents(topic_model, docs, topics, embeddings, reduced_embeddings, sample, hide_annotations, hide_document_hover, custom_labels, title, width, height)
102 indices = np.array(indices)
104 df = pd.DataFrame({"topic": np.array(topic_per_doc)[indices]})
--> 105 df["doc"] = [docs[index] for index in indices]
106 df["topic"] = [topic_per_doc[index] for index in indices]
108 # Extract embeddings if not already done

File [c:\Users\XXXAppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\bertopic\plotting\_documents.py:105](file:///C:/Users/XXXAppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/bertopic/plotting/_documents.py:105), in (.0)
102 indices = np.array(indices)
104 df = pd.DataFrame({"topic": np.array(topic_per_doc)[indices]})
--> 105 df["doc"] = [docs[index] for index in indices]
106 df["topic"] = [topic_per_doc[index] for index in indices]
108 # Extract embeddings if not already done

File [c:\Users\XXXAppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\pandas\core\frame.py:3896](file:///C:/Users/XXXAppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/pandas/core/frame.py:3896), in DataFrame.__getitem__(self, key)
3894 if self.columns.nlevels > 1:
3895 return self._getitem_multilevel(key)
-> 3896 indexer = self.columns.get_loc(key)
3897 if is_integer(indexer):
3898 indexer = [indexer]

File [c:\Users\XXX\AppData\Local\miniconda3\envs\csiro-horizon-scanning39\lib\site-packages\pandas\core\indexes\base.py:3797](file:///C:/Users/XXXAppData/Local/miniconda3/envs/csiro-horizon-scanning39/lib/site-packages/pandas/core/indexes/base.py:3797), in Index.get_loc(self, key)
3792 if isinstance(casted_key, slice) or (
3793 isinstance(casted_key, abc.Iterable)
3794 and any(isinstance(x, slice) for x in casted_key)
3795 ):
3796 raise InvalidIndexError(key)
-> 3797 raise KeyError(key) from err
3798 except TypeError:
3799 # If we have a listlike key, _check_indexing_error will raise
3800 # InvalidIndexError. Otherwise we fall through and re-raise
3801 # the TypeError.
3802 self._check_indexing_error(key)

KeyError: 413
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in bertopic/plotting/_documents.py at the type-checking code near lines 9-11, then follow BERTopic.visualize_documents to the failing access at line 105. Reproduce the provided pandas DataFrame example and make the affected methods report that docs must be a list instead of exposing a KeyError; verify the example produces the intended error.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-visualization, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.