MaartenGr / MaartenGr/BERTopic
Langchain Representation: No KEYWORDS tag included in generation
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi,
Why isn't there a KEYWORDS tag included in the langchain chain representation generation?
Can it be added? I think it can possibly improve label accuracy.
Something simple like this in _langchain.py?
# `self.chain` must take `input_documents` and `question` as input keys
# Use a custom prompt that leverages keywords, using the tag: [KEYWORDS]
if "[KEYWORDS]" in self.prompt:
prompts = []
for topic in topics:
keywords = list(zip(*topics[topic]))[0]
prompt = self.prompt.replace("[KEYWORDS]", ", ".join(keywords))
prompts.append(prompt)
inputs = [
{"input_documents": docs, "question": prompt}
for docs, prompt in zip(chain_docs, prompts)
]
else:
inputs = [
{"input_documents": docs, "question": self.prompt}
for docs in chain_docs
]
instead of:
inputs = [
{"input_documents": docs, "question": self.prompt}
for docs in chain_docs
]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in _langchain.py and inspect the chain representation generation around construction of the inputs passed to self.chain. Add support for a [KEYWORDS] prompt tag using the topic keywords, while preserving the existing behavior when the tag is absent; done means prompts can include those keywords without changing ordinary prompts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100