MaartenGr / MaartenGr/BERTopic
Topics over time: IndexError: index (68) out of range
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Looks related to this.
The environment:
Windows 64-bit
bertopic 0.17.3 pypi_0 pypi
hdbscan 0.8.40 pypi_0 pypi
numpy 2.2.6 pypi_0 pypi
python 3.13.7 hdf00ec1_100_cp313 conda-forge
sentence-transformers 5.1.0 pypi_0 pypi
umap-learn 0.5.9.post2 pypi_0 pypi
The code:
df = pd.read_csv(data_path)
documents = df['text'].tolist()
date_format = '%Y-%m-%d %H:%M:%S'
df['date'] = pd.to_datetime(df['date'], format=date_format)
df['date'] = df['date'].dt.date.astype(str)
timestamps = df['date'].tolist()
topic_model = BERTopic(verbose=True)
topics, probabilities = topic_model.fit_transform(documents)
topics_over_time = topic_model.topics_over_time(documents, topics, timestamps, nr_bins=20, global_tuning=True, evolution_tuning=True)
The error:
Traceback (most recent call last):
File "D:\Users\user\workspace\Customer-DS009_topic_modelling\src\run_bertopic_simple.py", line 64, in
topics_over_time = topic_model.topics_over_time(documents, topics, timestamps, nr_bins=20, global_tuning=True, evolution_tuning=True)
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\bertopic_bertopic.py", line 925, in topics_over_time
c_tf_idf = (global_c_tf_idf[selected_topics] + c_tf_idf) / 2.0
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 30, in getitem
index, new_shape = self._validate_indices(key)
~~~~~~~~~~~~~~~~~~~~~~^^^^^
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 288, in _validate_indices
idx = self._asindices(idx, N)
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 332, in _asindices
raise IndexError(f'index ({max_indx}) out of range')
IndexError: index (68) out of range
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the provided Windows environment and example code, starting at BERTopic's topics_over_time entry point and the _bertopic.py line 925 shown in the traceback. Trace how selected_topics is used with the global c-TF-IDF sparse matrix and compare it with the scipy sparse indexing error. Done means the example completes without the IndexError and produces the topics-over-time result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100