MaartenGr / MaartenGr/BERTopic

Topics over time: IndexError: index (68) out of range

Open
#2,425 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Looks related to this.

The environment:

Windows 64-bit

bertopic 0.17.3 pypi_0 pypi
hdbscan 0.8.40 pypi_0 pypi
numpy 2.2.6 pypi_0 pypi
python 3.13.7 hdf00ec1_100_cp313 conda-forge
sentence-transformers 5.1.0 pypi_0 pypi
umap-learn 0.5.9.post2 pypi_0 pypi

The code:

    df = pd.read_csv(data_path)
    documents = df['text'].tolist()
    date_format = '%Y-%m-%d %H:%M:%S'
    df['date'] = pd.to_datetime(df['date'], format=date_format)
    df['date'] = df['date'].dt.date.astype(str)
    timestamps = df['date'].tolist()

    topic_model = BERTopic(verbose=True)
    topics, probabilities = topic_model.fit_transform(documents)

    topics_over_time = topic_model.topics_over_time(documents, topics, timestamps, nr_bins=20, global_tuning=True, evolution_tuning=True)

The error:

Traceback (most recent call last):
File "D:\Users\user\workspace\Customer-DS009_topic_modelling\src\run_bertopic_simple.py", line 64, in
topics_over_time = topic_model.topics_over_time(documents, topics, timestamps, nr_bins=20, global_tuning=True, evolution_tuning=True)
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\bertopic_bertopic.py", line 925, in topics_over_time
c_tf_idf = (global_c_tf_idf[selected_topics] + c_tf_idf) / 2.0
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 30, in getitem
index, new_shape = self._validate_indices(key)
~~~~~~~~~~~~~~~~~~~~~~^^^^^
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 288, in _validate_indices
idx = self._asindices(idx, N)
File "C:\Users\user.conda\envs\bertopic\Lib\site-packages\scipy\sparse_index.py", line 332, in _asindices
raise IndexError(f'index ({max_indx}) out of range')
IndexError: index (68) out of range

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the provided Windows environment and example code, starting at BERTopic's topics_over_time entry point and the _bertopic.py line 925 shown in the traceback. Trace how selected_topics is used with the global c-TF-IDF sparse matrix and compare it with the scipy sparse indexing error. Done means the example completes without the IndexError and produces the topics-over-time result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.