IndexError: arrays used as indices must be of integer (or boolean) type

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Desribe the bug

Actually, I found similar, but closed issue so I would like to reopen that.

I'm trying to use this library for clustering news in stream-like way. So, when system is started it contains zero elements.

It's not clear how much elements do you need to be sure that clustering process should finish right.

I've lost several understand to ensure that this error (IndexError: arrays used as indices must be of integer (or boolean) type) PROBABLY occurs because there's few elements in topic. And I still continue to debug it.

Error says nothing about root cause and it would be nice to fix it...

Reproduction

import logging

import numpy as np
import redis.asyncio as redis
from minio import Minio
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic

from config.settings import settings
from umap import UMAP
import asyncio
from bertopic.representation import KeyBERTInspired


logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logging.basicConfig(
    level=logging.INFO,  # или INFO, если не нужны DEBUG-сообщения
    format="%(asctime)s %(levelname)s %(name)s %(message)s"
)

async def main():
    logger.info("Starting clustering service")
    r = redis.Redis(host=settings.redis.host, port=settings.redis.port, db=0)
    if not await r.ping():
        logger.error("Failed to connect to redis")
        exit(1)
    else:
        logger.info("Connected to redis")

    s3 = Minio(
        settings.minio.endpoint,
        access_key=settings.minio.access_key,
        secret_key=settings.minio.secret_key,
        secure=False,
    )

    pubsub = r.pubsub()

    await pubsub.subscribe("inbrief")

    model = SentenceTransformer(
        settings.embedding_model.model_name,
        trust_remote_code=settings.embedding_model.trust_remote_code
    )


    dim = model.encode("Hello world!", task="separation").shape[0]

    logger.debug("Embedding dimension: %s", dim)

    embeddings = np.empty((0, dim))
    texts = []

    umap = UMAP(n_neighbors=15, n_components=2, metric='cosine')
    bertopic = BERTopic(
        language=settings.embedding_model.language,
        embedding_model=model,
        representation_model=KeyBERTInspired()
    )


    while True:
        msg = await pubsub.get_message(timeout=None)
        if msg is None:
            break

        logger.debug("Received message: %s", msg)
        if msg["type"] != "message":
            continue


        filename = f"{msg['data'].decode('utf-8')}.json"

        resp = s3.get_object("inbrief", filename)

        payload = resp.json()

        new_texts = list(map(lambda x: x['text'], payload))
        new_embeddings = model.encode(new_texts, task="separation")
        embeddings = np.append(embeddings, new_embeddings, axis=0)
        texts.extend(new_texts)

        try:
            reduced_embeddings = umap.fit_transform(embeddings)
            bertopic.fit_transform(texts, embeddings=embeddings)

            bertopic.visualize_topics().write_html("static/topics.html")
            bertopic.visualize_documents(texts, reduced_embeddings=reduced_embeddings).write_html("static/documents.html")
        except Exception as e:
            logger.warning(f"Got an error while clustering, will try later: {e}", exc_info=True)
            continue

        logger.debug("Number of texts: %s", len(texts))



if __name__ == "__main__":
    asyncio.run(main())

BERTopic Version

0.17.0

Start by running the provided reproduction with an empty or very small embeddings array and inspect the BERTopic fit_transform path where the IndexError occurs. Compare behavior as texts are added and determine whether the failure is caused by insufficient documents. Done means the root cause is identified and the error is handled or reported clearly.