huggingface / huggingface/llm-ls

Proposal: Launching LLM server as a daemon

Open
#89 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
881
Forks
71
Avg merge
33m
Merged PRs (30d)
1

Description

Proof-of-concept: https://github.com/blmarket/llm-ls

Hi,

I'm wondering llm-ls can incorporate dedicated LLM server provider within LSP server, preferably as a shared instance via [daemonize](https://crates.io/crates/daemonize). Idea was inspired by [bazel client/server](https://bazel.build/run/client-server).

It works as following:

* There's a lock file to ensure only 1 LLM server is running
* If there's no usage of LLM (i.e. LSP exits) for 10 seconds, LLM server is terminated
* There is a heartbeat to let LLM process know there is a client
* Multiple LSP instances will share single LLM server

Still many things are hardcoded (such as LLM path, various file path, model path), but it's usable with existing llm.XXX editor plugins.

Want to know whether this is something llm-ls would like to support as one of its backends, or better be kept as different project.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.