microsoft / microsoft/LLMLingua

Getting errors when running phi2

Open
#78 1 comment 0 reactions 1 assignee View on GitHub

@iofu728 is already working on this.

Since Jan 31, 2024.

question
Dominant language
Python
Stars
6.7k
Forks
428
Avg merge
2d 4h
Merged PRs (30d)
1

Description

I'm getting the following error:

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|██████████| 2/2 [00:02<00:00,  1.15s/it]
Some weights of the model checkpoint at ../compressor/compressor_llm/phi2_dolphin were not used when initializing PhiForCausalLM: ['lm_head.linear.lora_A.default.weight', 'lm_head.linear.lora_B.default.weight']
- This IS expected if you are initializing PhiForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing PhiForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
  File "/home/tempus/assistant/pythonProject/llm.py", line 54, in <module>
    original_prompt = create_prompt(system=SYSTEM_MESSAGE, template=PROMPT_TEMPLATE, context_template=CONTEXT_TEMPLATE, user_prompt=prompt_input)
  File "/home/tempus/assistant/pythonProject/llm.py", line 31, in create_prompt
    context = token_compressor.compress_context(user_prompt, context)
  File "/home/tempus/assistant/pythonProject/token_compressor.py", line 16, in compress_context
    compressed_context = compressor.compress_prompt(
  File "/home/tempus/miniconda3/envs/llm/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 253, in compress_prompt
    context = self.iterative_compress_prompt(
  File "/home/tempus/miniconda3/envs/llm/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 754, in iterative_compress_prompt
    past_key_values = [
TypeError: 'NoneType' object is not iterable

It works just fine with a small amount of tokens (<~350) and throws out this error when I give it some more tokens.
Am I doing it wrong? It doesn't happen with other models

This is my code:

from llmlingua import PromptCompressor

instruction = "summarize the following text. Keep key information. Do not add any additional data and keep the facts accurate."


def compress_context(prompt, context):

    context_list = context.split("\n")
    context_list = ["\n".join(context_list[ii: ii + 4]) for ii in range(0, len(context_list), 4)]

    compressor = PromptCompressor(model_name="../compressor/compressor_llm/phi2_dolphin")

    compressed_context = compressor.compress_prompt(
        context=context_list,
        instruction=instruction,
        question=prompt,
        condition_compare=True,
        condition_in_question='after',
        rank_method='longllmlingua',
        use_sentence_level_filter=False,
        dynamic_context_compression_ratio=0.4,  # enable dynamic_context_compression_ratio
        ratio=0.5,
        concate_question=False
    )

    return compressed_context["compressed_prompt"]

I'm using the dolphin finetune for this

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.