Feature request: Huggingface tokenizer support
Aperta
- Lingua principale
- Rust
- Stelle
- 619
- Fork
- 74
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Since world tokenizer training code is not available as far as I know, those of us who need a custom tokenizer train HF tokenizer (pip rwkv package, RWKV-LM trainer and json2binidx_tool all support it).
Currently it doesn't work with ai00_server:
> [ai00_server::middleware] reload model failed: failed to parse vocabulary: invalid value: expected key to be a number in quotes at line 2 column 3
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.