MaartenGr / MaartenGr/BERTopic

Add structured multi aspect output to the LangChain representation 2:06 AM

Open
#2,523 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Feature request

I propose modernizing the existing LangChain representation interface and adding native structured multi aspect output. The updated interface would accept custom system and user prompts while BERTopic internally prepares and supplies keywords and representative documents. Users could declare an output schema as a mapping of field names to descriptions. Each topic would then produce one validated JSON response containing fields such as label and description.

A designated main field would update `topic_representations_`, while the remaining fields would populate `topic_aspects_` and appear as columns in `get_topic_info()`. Existing features such as Runnable support, representative document selection, truncation, diversity, batch execution, and configurable concurrency would be preserved.

### Motivation

I am an algorithm engineer currently building a topic discovery project where BERTopic is an important part of our technical stack. Through intensive use with a large volume of data, we found that generating multiple topic aspects is less efficient and flexible than it could be.

The current LangChain representation supports batch execution, but expects each result to contain a single `output_text` value. Multi aspect representations are executed separately. For example, generating a label and description for 100 topics requires 200 model requests. Returning both fields from one structured response would reduce this to 100 requests. This avoids repeated input context, reduces latency and rate limit pressure, and improves consistency because all fields are generated from the same documents, keywords, and model response.

Custom system and user prompts combined with a declared output schema would also make the interface more flexible and allow users to obtain application ready results with less configuration. This proposal is related to #2187 and #2188. Those discussions explore updating LangChain support and structured output, but the current implementation does not preserve multiple named fields or map one response into multiple BERTopic aspects.

### Your contribution

I am willing to implement this feature. I can update the existing LangChain representation while preserving backward compatibility and its current batch and concurrency capabilities. I plan to add custom system and user prompts, structured output validation, support for multiple named fields, and integration with `topic_representations_` and `topic_aspects_`.

I will also add unit tests and documentation covering the existing single label behavior, structured multi field output, custom prompts, concurrency, invalid responses, and compatibility with custom LangChain Runnables. I would appreciate feedback on the proposed interface and return protocol before beginning the implementation.

:)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing LangChain representation and review the related discussions in #2187 and #2188 before proposing the interface. Done means preserving current single-label, batching, concurrency, Runnable, and document-selection behavior while supporting validated multi-field responses that update topic_representations_, topic_aspects_, and get_topic_info(), with tests and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.