github / github/rust-gems

`bpe` Python Bindings?

Open
#61 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
134
Forks
24
Avg merge
16h 27m
Merged PRs (30d)
11

Description

Amazing work on `bpe`!

Wanted to see if there were any known Python bindings a la the snippets from tiktoken below?

```python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
```

Contributor guide

Open the contributing guide

Research direction

The issue names no implementation files or tests. Start by reviewing the bpe implementation and its public API, then determine whether Python bindings already exist and what interface is needed to support the shown encoding and decoding examples. Done would require an agreed binding scope and corresponding Python usage documentation and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.