AbrahamSanders / AbrahamSanders/codec-bpe

compression rate

Abierto
#2 6 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
77
Forks
8
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

good work!
I'm using 10M audio codecs to train codec-bpe with a `vocab_size=30k,num_codebook=4,codebook_size=1024`, after training I get a tokenizer and found the compression rate is nearly 1. For example, a 4s audio with 25hz, I get 400 audio codec tokens, and using codec-bpe I get about nearly the same token-size about more than 350 which is less then 1 ,`this can yield savings of 2-5x in sequence length compared to directly modeling the flattened codebooks `. I follow the steps in the readme, so can you share your codec-bpe tokenizer based on encodec or some other codec?

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.