alexanderatallah / alexanderatallah/window.ai
Using basaran for loading custom Hugging Face models
- Vorherrschende Sprache
- TypeScript
- Sterne
- 986
- Forks
- 97
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Just adding my notes here on how to run `basaran` as a local backend to Window. Happy to add the below to wherever appropriate in the repo.
Their [repo](https://github.com/hyperonym/basaran) makes it pretty easy to set up with just a python virtualenv. The process for me was as simple as:
## Process
1. Create a virtualenv with `virtualenv -p python3 basaran_env` and activate it `source basaran_env/bin/activate`.
2. `pip install basaran`
3. Then, running `MODEL=user/repo PORT=8000 python -m basaran` downloads the model located at `https://huggingface.co//`, (e.g., `gpt2` or `allenai/tk-instruct-3b-def`) to the current folder and serve it via `localhost:8000/v1/completions`. You can confirm it works by running:
```
curl http://127.0.0.1/v1/completions \
-H 'Content-Type: application/json' \
-d '{ "prompt": "once upon a time," }'
```
4. Set the `Local` model endpoint in Window to `http://127.0.0.1:8000/v1` (instead of the default `http://127.0.0.1:8000/`).
## Notes
- Supposedly, it also supports streaming.
- It's not working with the Windowai.io home page or the [ChatbotUI](https://www.skylightai.io/board/06df28e9-a416-4400-8c06-107ef6a51780), which I suspect is because they expect a chat endpoint instead of the given completions endpoint, (although it does work for [Jema](https://www.skylightai.io/board/9600d425-933c-446c-9e71-8f381a7de9fe) and [Hangman](https://www.skylightai.io/board/34354771-c3e0-4c06-86c7-c754d42d8f38)).
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.