ollama / ollama/ollama-python

Unable to generate responses normally when invoking the fine-tuned model using Python.

Open
#288 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.5k
Forks
1.2k
Avg merge
4m
Merged PRs (30d)
1

Description

Body: Greetings : ),

After invoking the llamafactory fine-tuned qwen2-7B model using ollama.chat(), the model is unable to recognize the system prompt.
The model sometimes fails to generate any response when prompted with other input phrases.

Here are two scenarios that illustrate the issues:
code

import ollama

model_list = ['qwen2', 'glm4', 'lawdamo2']
model = model_list[2]  #Here is my own model

def LLM_Process(model, sys_prom, usr_prom):
    messages = [
        {'role': 'user', 'content': usr_prom},
        {'role': 'system', 'content': sys_prom}
    ]
    options = {
        'temperature': 0.1
    }
    resp = ollama.chat(model, messages, options=options)
    print(resp)


LLM_Process(model, 'You are a Dark Tyrannosaur War God', 'Design a stylish slogan for yourself.')

output

{'model': 'lawdamo2', 'created_at': '2024-09-23T06:50:51.4908085Z', 'message': {'role': 'assistant', 'content': 'As an AI language model, I don\'t have a physical appearance or personal style to showcase, but here\'s a suggestion for a slogan that could represent the essence of my capabilities:\n\n"Unleashing intelligence, empowering knowledge - your ultimate cognitive companion."'}, 'done_reason': 'stop', 'done': True, 'total_duration': 9875967000, 'load_duration': 8973791800, 'prompt_eval_count': 19, 'prompt_eval_duration': 25338000, 'eval_count': 51, 'eval_duration': 871361000}

code

import ollama
from tqdm import tqdm
import pandas as pd

model_list = ['qwen2', 'glm4', 'lawdamo2']
model = model_list[2]  #Here is my own model

def LLM_Process(model, sys_prom, usr_prom):
    messages = [
        {'role': 'user', 'content': usr_prom},
        {'role': 'system', 'content': sys_prom}
    ]
    options = {
        'temperature': 0.1
    }
    resp = ollama.chat(model, messages, options=options)
    print(resp)


inputdir = './data/crime_data.csv'
# Assuming the output directory has a proper format to include line numbers
outputdir = './output/processed_data_{start_line}_{end_line}.txt'

sysP = 'As a criminal geographer, please analyze the legal document I provide to you and output the addresses where the crimes occurred, separated by line breaks if there are multiple addresses. If no detailed address information is provided in the document or if you are unable to discern the addresses, simply output NaN. Do not reply with any content other than the addresses of the crimes. Thank you for your cooperation!'
allin = pd.read_csv(inputdir)
keys = ['UUID', '正文']
usrP = []
for index, row in tqdm(allin.iterrows(), total=allin.shape[0]):
    content1 = row[keys[0]]
    content2 = row[keys[1]]
    usrP.append(content2)


LLM_Process(model, sysP, usrP[0])

output

{'model': 'lawdamo2', 'created_at': '2024-09-23T06:57:19.4138535Z', 'message': {'role': 'assistant', 'content': ''}, 'done_reason': 'stop', 'done': True, 'total_duration': 3629228100, 'load_duration': 3422628100, 'prompt_eval_count': 1304, 'prompt_eval_duration': 201533000, 'eval_count': 1, 'eval_duration': 13000}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Python ollama.chat() entry point and reproduce both snippets with the listed models, especially lawdamo2. Check whether the system-prompt behavior and empty response can be reproduced; done means a confirmed client-side issue with a focused failing case, or evidence that the behavior depends on the fine-tuned model.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.