alexanderatallah / alexanderatallah/window.ai

Using basaran for loading custom Hugging Face models

Offen
#45 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
TypeScript
Sterne
986
Forks
97
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Just adding my notes here on how to run `basaran` as a local backend to Window. Happy to add the below to wherever appropriate in the repo.

Their [repo](https://github.com/hyperonym/basaran) makes it pretty easy to set up with just a python virtualenv. The process for me was as simple as:

## Process
1. Create a virtualenv with `virtualenv -p python3 basaran_env` and activate it `source basaran_env/bin/activate`.
2. `pip install basaran`
3. Then, running `MODEL=user/repo PORT=8000 python -m basaran` downloads the model located at `https://huggingface.co//`, (e.g., `gpt2` or `allenai/tk-instruct-3b-def`) to the current folder and serve it via `localhost:8000/v1/completions`. You can confirm it works by running:

```
curl http://127.0.0.1/v1/completions \
-H 'Content-Type: application/json' \
-d '{ "prompt": "once upon a time," }'
```

4. Set the `Local` model endpoint in Window to `http://127.0.0.1:8000/v1` (instead of the default `http://127.0.0.1:8000/`).

## Notes
- Supposedly, it also supports streaming.
- It's not working with the Windowai.io home page or the [ChatbotUI](https://www.skylightai.io/board/06df28e9-a416-4400-8c06-107ef6a51780), which I suspect is because they expect a chat endpoint instead of the given completions endpoint, (although it does work for [Jema](https://www.skylightai.io/board/9600d425-933c-446c-9e71-8f381a7de9fe) and [Hangman](https://www.skylightai.io/board/34354771-c3e0-4c06-86c7-c754d42d8f38)).

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.