MaartenGr / MaartenGr/BERTopic

Langchain Representation: No KEYWORDS tag included in generation

Open
#1,855 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi,

Why isn't there a KEYWORDS tag included in the langchain chain representation generation?

Can it be added? I think it can possibly improve label accuracy.

Something simple like this in _langchain.py?

        # `self.chain` must take `input_documents` and `question` as input keys
        # Use a custom prompt that leverages keywords, using the tag: [KEYWORDS]
        if "[KEYWORDS]" in self.prompt:
            prompts = []
            for topic in topics:
                keywords = list(zip(*topics[topic]))[0]
                prompt = self.prompt.replace("[KEYWORDS]", ", ".join(keywords))
                prompts.append(prompt)

            inputs = [
                {"input_documents": docs, "question": prompt}
                for docs, prompt in zip(chain_docs, prompts)
            ]
            
        else:
            inputs = [
                {"input_documents": docs, "question": self.prompt}
                for docs in chain_docs
            ]

instead of:

            inputs = [
                {"input_documents": docs, "question": self.prompt}
                for docs in chain_docs
            ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in _langchain.py and inspect the chain representation generation around construction of the inputs passed to self.chain. Add support for a [KEYWORDS] prompt tag using the topic keywords, while preserving the existing behavior when the tag is absent; done means prompts can include those keywords without changing ordinary prompts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.