KeyLLM keyword extraction issue
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 385
- PR merge metrics
- No merged PRs in 30d
Description
KeyLLM seems to be extracting keywords which are not even present in the document used. I am following the steps mentioned in this article - https://towardsdatascience.com/introducing-keyllm-keyword-extraction-with-llms-39924b504813
I am using Mistral 7B model.
```
from ctransformers import AutoModelForCausalLM
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
model = AutoModelForCausalLM.from_pretrained(
"TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
model_type="mistral",
gpu_layers=50,
hf=True
)
from transformers import AutoTokenizer, pipeline
# Tokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
# Pipeline
generator = pipeline(
model=model, tokenizer=tokenizer,
task='text-generation',
max_new_tokens=50,
repetition_penalty=1.1
)
```
```
from keybert.llm import TextGeneration
from keybert import KeyLLM
# Load it in KeyLLM
llm = TextGeneration(generator, prompt=prompt)
kw_model = KeyLLM(llm)
documents = [
"As discussed above, for the training set, finer-grained instances in the training set are generally better than coarser-grained ones. This preference does not apply to classification time, i.e. the use of the classifier in the field. We should go ahead and predict the sentiment of whatever text we are given, be it a sentence or a chapter.",
"I received my package!",
"You clearly want to know what is being complained about and what is being liked."
]
keywords = kw_model.extract_keywords(documents); keywords
```
**Output -**
```
[['discussed',
'above',
'finer-grained',
'instances',
'training',
'set',
'better',
'coarser-grained',
'preference',
'applies',
'classification',
'time',
'field',
'predict',
'sentiment',
'text',
'sentence',
'chapter.'],
['package',
'received',
'delivery',
'shipment',
'mail',
'courier',
'product',
'order',
'online',
'store'],
['complained',
'liked',
'want',
'know',
'clear',
'understand',
'specific',
'detail',
'issue',
'problem',
'feedback',
'opinion',
'satisfaction',
'enjoyment',
'appreciation',
'preference',
'dislike',
'dissatisfaction',
'negative',
'positive',
'favorable',
'unf']]
```
It seems to be extracting similar words even though they are not present in the original document. Seems like model specific issue?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the example with the shown Python setup, then inspect the KeyLLM TextGeneration path and compare the generated keywords with the source documents. Done means determining whether the behavior comes from the model or KeyLLM and documenting a reproducible conclusion or scoped fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100