`bpe` Python Bindings?
- Dominant language
- Rust
- Stars
- 134
- Forks
- 24
- Avg merge
- 16h 27m
- Merged PRs (30d)
- 11
Description
Amazing work on `bpe`!
Wanted to see if there were any known Python bindings a la the snippets from tiktoken below?
```python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
```
Contributor guide
Research direction
The issue names no implementation files or tests. Start by reviewing the bpe implementation and its public API, then determine whether Python bindings already exist and what interface is needed to support the shown encoding and decoding examples. Done would require an agreed binding scope and corresponding Python usage documentation and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100