MaartenGr / MaartenGr/BERTopic

The 'X' parameter of normalize must be an array-like or a sparse matrix.

Open
#2,343 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [x] I have searched and found no existing issues

### Desribe the bug

I have a very large dataset, so I have been saving and loading models as is documented a good practice elsewhere. The problem is, when I load and merge these, I cannot do the class-based or time-dependent topic representations, as it seems c-tf-idf information is not saved when a model is saved to disk in the model.

Full error log:

```
Traceback (most recent call last):
File "/Users/ysc4337/warlock/merging-test/main.py", line 56, in
topics_over_time = merged_model.topics_over_time(all_docs, all_timestamps)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ysc4337/warlock/merging-test/.venv/lib/python3.11/site-packages/bertopic/_bertopic.py", line 860, in topics_over_time
global_c_tf_idf = normalize(self.c_tf_idf_, axis=1, norm="l1", copy=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ysc4337/warlock/merging-test/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py", line 206, in wrapper
validate_parameter_constraints(
File "/Users/ysc4337/warlock/merging-test/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py", line 98, in validate_parameter_constraints
raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'X' parameter of normalize must be an array-like or a sparse matrix. Got None instead.
(merging-test) ➜ merging-test git:(master) ✗
```

I have tried saving the c_tf_idf matrix as a workaround, but I could not get the code working.

### Reproduction

```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
import random
from datetime import datetime, timedelta

# Fetch the dataset
data = fetch_20newsgroups(subset="all")
documents = data.data
categories = data.target

# Generate synthetic timestamps
start_date = datetime(2000, 1, 1)
timestamps = [
str(start_date + timedelta(days=random.randint(0, 7300)))
for _ in range(len(documents))
]

# Combine data with timestamps and categories
data_with_metadata = list(zip(documents, timestamps, categories))

# Split data into two parts for training two separate models
data_part1 = data_with_metadata[: len(data_with_metadata) // 2]
data_part2 = data_with_metadata[len(data_with_metadata) // 2 :]

# Extract documents and timestamps from both parts
docs_part1 = [doc for doc, _, _ in data_part1]
docs_part2 = [doc for doc, _, _ in data_part2]
timestamps_part1 = [ts for _, ts, _ in data_part1]
timestamps_part2 = [ts for _, ts, _ in data_part2]

model1 = BERTopic(verbose=True)
model2 = BERTopic(verbose=True)

topics, _ = model1.fit_transform(docs_part1)
topics, _ = model2.fit_transform(docs_part2)

model1.save("bertopic_model1")
model2.save("bertopic_model2")

model1 = BERTopic.load("bertopic_model1")
model2 = BERTopic.load("bertopic_model2")
# Merge the models

merged_model = BERTopic.merge_models([model1, model2])

# Reload the models

# Combine all documents and timestamps
all_docs = docs_part1 + docs_part2
all_timestamps = timestamps_part1 + timestamps_part2

# Fit the merged model to generate the c-TF-IDF matrix

# Generate topics over time visualization
topics_over_time = merged_model.topics_over_time(all_docs, all_timestamps)
merged_model.visualize_topics_over_time(topics_over_time).write_html(
"topics_over_time.html"
)
```

### BERTopic Version

0.17.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with bertopic/_bertopic.py, especially BERTopic.save, load, merge_models, and topics_over_time, then run the provided 20 Newsgroups reproduction. Trace why merged_model.c_tf_idf_ is None after loading and merging; done means the reproduced topics_over_time call no longer raises the normalize error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.