Error running llamafile: 'Choice' object has no attribute 'logprobs'

I've noticed that when streaming from llamafiles via llm with a config as in https://simonwillison.net/2023/Dec/18/mistral/#llamafile-openai that the response is suffixed with Error: 'Choice' object has no attribute 'logprobs'. It looks like the v1 API doesn't include those fields. It doesn't affect the response but it is kind of distracting.

extra-openai-models.yaml (I'm always running something on 8080 so it's running on 8081):

- model_id: llamafile
  model_name: llamafile
  api_base: "http://127.0.0.1:8081/v1"

Usage:

llm 'Hello world' -m llamafile
Hello userError: 'Choice' object has no attribute 'logprobs'

With verbose logging:

LLM_OPENAI_SHOW_RESPONSES=true llm 'Hello world' -m llamafile
Request: POST http://127.0.0.1:8081/v1/chat/completions
  Headers:
    host: 127.0.0.1:8081
    connection: keep-alive
    accept: application/json
    content-type: application/json
    user-agent: OpenAI/Python 1.3.8
    x-stainless-lang: python
    x-stainless-package-version: 1.3.8
    x-stainless-os: MacOS
    x-stainless-arch: arm64
    x-stainless-runtime: CPython
    x-stainless-runtime-version: 3.11.6
    authorization: [...]
    x-stainless-async: false
    content-length: 96
  Body:
    {
      "messages": [
        {
          "role": "user",
          "content": "Hello world"
        }
      ],
      "model": "llamafile",
      "stream": true
    }
Response: status_code=200
  Headers:
    access-control-allow-origin:
    content-type: text/event-stream
    keep-alive: timeout=5, max=5
    server: llama.cpp
    transfer-encoding: chunked
  Body:
    data: {"choices":[{"delta":{"content":"Hello"},"finish_reason":null,"index":0}],"created":1708025779,"id":"chatcmpl-akzf3K0XmOlPWWw1w1OcDLQfzYjnHGRh","model":"llamafile","object":"chat.completion.chunk"}


Hello    data: {"choices":[{"delta":{"content":" user"},"finish_reason":null,"index":0}],"created":1708025779,"id":"chatcmpl-SYecAQrW2AqWw2ypmqASMEO2QCSpdShs","model":"llamafile","object":"chat.completion.chunk"}


 user    data: {"choices":[{"delta":{},"finish_reason":"stop","index":0}],"created":1708025779,"id":"chatcmpl-wspOidmnVzxAP2nzjSaoiWmXhsFQFk8K","model":"llamafile","object":"chat.completion.chunk"}


Error: 'Choice' object has no attribute 'logprobs'

It doesn't affect --no-stream responses.

I've observed the behavior with these llamafiles listed on the llamafile repo:

• https://huggingface.co/jartine/phi-2-llamafile/resolve/main/phi-2.Q5_K_M.llamafile?download=true
• https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4.llamafile?download=true

Reproduce the streaming request with the llamafile configuration in extra-openai-models.yaml and the llm command, then trace the streamed Choice handling that accesses logprobs. The fix is done when streaming completes without the attribute error while --no-stream behavior remains unchanged; add or update a regression test if the existing test suite covers this path.