lm-sys / lm-sys/FastChat

Is the OpenAI-compatible API sending the `functions` argument to the model?

Open
#2,425 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I'm currently using the OS model functionary, which supports functions in a manner similar to how GPT operates through the OpenAI API. I've successfully deployed the model worker and proceeded to make chat completion requests using the functions feature:

curl {API}/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "functionary-7b",
    "messages": [{"role": "user", "content": "salute Anthony"}],
  "functions": [
    {"name": "salute",
    "description": "This function can be used to salute someone.",
    "parameters": {
      "properties": {
        "who": {
          "description": "Name of whom to salute. o7",
          "title": "Who",
          "type": "string"
        }
      },
      "required": [
        "who"
      ],
      "title": "salute",
      "type": "object"
    }
  }]
  }'

The expected outcome was to receive a message that included the function_call argument, indicating the use of the salute function with the argument Anthony. However, the actual result was as follows:

{"id":"chatcmpl-fUcuEKMeCkavtjsPCzeKft","object":"chat.completion","created":1694683275,"model":"functionary-7b","choices":[{"index":0,"message":{"role":"assistant","content":"\nSalute!\n"},"finish_reason":"stop"}],"usage":{"prompt_tokens":579,"total_tokens":585,"completion_tokens":6}}

It appears that the functions argument is being ignored altogether. My question is whether fastchat/serve/openai_api_server.py is indeed sending the functions argument to the model.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fastchat/serve/openai_api_server.py and trace how the OpenAI-compatible chat completion request handles the functions field. Reproduce the provided curl request against the functionary-7b model, then verify whether functions reaches the model and whether the response includes the expected function_call data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.