epfl-dlab / epfl-dlab/transformers-CFG
remove legacy TokenTrie
Open
- Dominant language
- Python
- Stars
- 139
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
Currently there is ByteTrie at https://github.com/epfl-dlab/transformers-CFG/blob/main/transformers_cfg/tokenization/trie.py#L21
and also a TokenTrie at https://github.com/epfl-dlab/transformers-CFG/blob/main/transformers_cfg/tokenization/vocab_struct.py#L13
TokenTrie is the legacy code from previous implementation and it should be completely removed and replaced by ByteTrie.
There is still dependency on it in the parsing, so may need to treat it carefully
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.