huggingface / huggingface/tokenizers

Proposal: Add Golang Bindings for tokenizers

Open
#1,751 4 comments 27 reactions 0 assignees View on GitHub
Feature Request
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

- **Performance-Critical Applications** – Many performance-intensive apps use Go for concurrency and efficiency. Native bindings allow seamless integration without overhead.
- **Server-Side NLP** – Go is widely used in backend services. Golang bindings would make it easier to use Hugging Face tokenizers in APIs, microservices, and inference systems.
- **Better Compatibility** – Many Go-based projects rely on Python/C++ workarounds for NLP tasks. Direct bindings would reduce complexity and improve developer experience.
- **Improved Inference Pipelines** – Many real-time AI systems are built in Go. Native tokenizers would remove Python bottlenecks, making pipelines faster and more reliable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.