MaartenGr / MaartenGr/KeyBERT

KeyLLM keyword extraction issue

Open
#183 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

KeyLLM seems to be extracting keywords which are not even present in the document used. I am following the steps mentioned in this article - https://towardsdatascience.com/introducing-keyllm-keyword-extraction-with-llms-39924b504813

I am using Mistral 7B model.

```
from ctransformers import AutoModelForCausalLM

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
model = AutoModelForCausalLM.from_pretrained(
"TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
model_type="mistral",
gpu_layers=50,
hf=True
)

from transformers import AutoTokenizer, pipeline

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

# Pipeline
generator = pipeline(
model=model, tokenizer=tokenizer,
task='text-generation',
max_new_tokens=50,
repetition_penalty=1.1
)
```

```
from keybert.llm import TextGeneration
from keybert import KeyLLM

# Load it in KeyLLM
llm = TextGeneration(generator, prompt=prompt)
kw_model = KeyLLM(llm)

documents = [
"As discussed above, for the training set, finer-grained instances in the training set are generally better than coarser-grained ones. This preference does not apply to classification time, i.e. the use of the classifier in the field. We should go ahead and predict the sentiment of whatever text we are given, be it a sentence or a chapter.",
"I received my package!",
"You clearly want to know what is being complained about and what is being liked."
]

keywords = kw_model.extract_keywords(documents); keywords
```

**Output -**

```
[['discussed',
'above',
'finer-grained',
'instances',
'training',
'set',
'better',
'coarser-grained',
'preference',
'applies',
'classification',
'time',
'field',
'predict',
'sentiment',
'text',
'sentence',
'chapter.'],
['package',
'received',
'delivery',
'shipment',
'mail',
'courier',
'product',
'order',
'online',
'store'],
['complained',
'liked',
'want',
'know',
'clear',
'understand',
'specific',
'detail',
'issue',
'problem',
'feedback',
'opinion',
'satisfaction',
'enjoyment',
'appreciation',
'preference',
'dislike',
'dissatisfaction',
'negative',
'positive',
'favorable',
'unf']]
```

It seems to be extracting similar words even though they are not present in the original document. Seems like model specific issue?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the example with the shown Python setup, then inspect the KeyLLM TextGeneration path and compare the generated keywords with the source documents. Done means determining whether the behavior comes from the model or KeyLLM and documenting a reproducible conclusion or scoped fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.