Ai00-X / Ai00-X/ai00_server

Server Startup Issue

Aberta
#110 5 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Rust
Estrelas
619
Forks
74
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

Hi All,

I am trying to run `cargo run --release` and I get the following:
Finished `release` profile [optimized] target(s) in 0.21s
Running `target/release/ai00_server`
2024-05-04T04:35:17.007Z INFO [ai00_server] reading config assets/configs/Config.toml...
2024-05-04T04:35:17.008Z INFO [ai00_server::middleware] ModelInfo {
version: V5,
num_layer: 32,
num_emb: 2560,
num_hidden: 8960,
num_vocab: 65536,
num_head: 40,
time_mix_adapter_size: 0,
time_decay_adapter_size: 0,
}
2024-05-04T04:35:17.008Z INFO [ai00_server::middleware] type: SafeTensors
2024-05-04T04:35:17.013Z WARN [wgpu_hal::gles::egl] EGL_MESA_platform_surfaceless not available. Using default platform
2024-05-04T04:35:17.027Z WARN [wgpu_hal::gles::egl] No config found!
2024-05-04T04:35:17.027Z WARN [wgpu_hal::gles::egl] No config found!
2024-05-04T04:35:17.048Z INFO [ai00_server] server started at 0.0.0.0:65530 with tls
2024-05-04T04:35:17.071Z ERROR [ai00_server::middleware] reload model failed: failed to request device

Currently, it is Cpu ins Config.toml. Also, I have Nvidia 22C (24GB) GPU and it gives the same error when I set it to Gpu.
Result of `lsb_release -a`:

```
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

```

Config.toml:
```
[model]
embed_device = "Cpu" # Device to put the embed tensor ("Cpu" or "Gpu").
max_batch = 8 # The maximum batches that are cached on GPU.
model_name = "RWKV-5-World-3B-v2-20231113-ctx4096.st" # Name of the model.
model_path = "assets/models" # Path to the folder containing all models.
quant = 32 # Layers to be quantized.
quant_type = "Int8" # Quantization type ("Int8" or "NF4").
stop = ["\n\n"] # Additional stop words in generation.
token_chunk_size = 128 # Size of token chunk that is inferred at once. For high end GPUs, this could be 64 or 128 (faster).

```
Can anyone help? Thanks

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.