Proxy using SOCKS5, and openai base prompt as a list

Validations

• I believe this is a way to improve. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that requests the same enhancement

Problem

My institution is offering a locally hosted model. The following example is a simple prompt request with curl outside the local network using SSH tunneling ssh -D 8080 server.xyz :

curl -X POST \
     --socks5 localhost:8080 \
     -H "Content-Type: application/json" \
      -d @- \
    https://apps.inside.xyz/argoapi/api/v1/resource/chat/ << EOF
{
"user" : "REDACTED",
"model": "gpt35",
"prompt": ["Why is the sky blue?"]
}
EOF

I believe that this falls in the category of OpenAI compatible servers.

If I want to add this to the models in Continue.dev, how the conf dictionary would look like?

Maybe something like this?

    {
      "title": "argo-gpt",
      "model": "gpt35",
      "user" : "myuser",
      "provider": "openai",
      "apiKey": "EMPTY",
      "apiBase": "https://apps.inside.anl.gov/argoapi/api/v1/resource/",
      "requestOptions": {
        "proxy":"socks5://localhost:8080"
      }
    }

Also it is intriguing that the prompt should be as a list...

Thanks

Solution

No response